Install cronjobs

Install cronjobs and scripts

Install scripts for to.science

The scripts constitute a set of helpful tools about the administration of a to.science installation.
Most of the scripts have been written for the use in cronjobs, such as
– backups of database, of the search index
– re-indexing the search index
– nightly web crawling
– update OAI sets
– export webserver logs to a statistics tool
– and more
They can also be run on demand from Linux console.
Most scripts have been written in Linux bash (Bourne Again Shell) scripting language.

Install the scripts:

cd /opt/toscience/ 
  git clone "https://github.com/hbz/to.science.scripts.git" bin 
  cd bin 
  ln -s ../git/to.science.install/variables.conf

The latter command line adds the link to the global configuration file of to.science. The configuration file is needed by almost all scripts to run.

Install cronjobs for to.science

Install basic cronjobs for a toscience instance that has webcrawling facilities:

crontab -e

# m h  dom mon dow   command
# Update Metadata (lobdify & enrich)
0 5 * * 6 /opt/toscience/bin/updateAll.sh > /dev/null
# Anonymize local apache logs
0 1 * * * /opt/toscience/bin/depersonalize-apache-logs.sh > /dev/null
# Load apache logs of the previous day into the local piwik instance
0 2 * * * /opt/toscience/bin/import-logfiles.sh > /dev/null
# Run Edoweb Webgathering
0 20 * * * /opt/toscience/bin/runGatherer.sh >> /opt/toscience/logs/runGatherer.log
# Evaluate the latest Webgatherer run
0 21 * * * /opt/toscience/bin/evalWebgatherer.sh >> /opt/toscience/logs/runGatherer.log
# Move files out of the working directory of wpull into the output directory of wpull
0 22 * * * /opt/toscience/bin/move_files_from_crawldir.sh >> /opt/toscience/logs/ks.move_files_from_crawldir.log
# Generate Crawl Reports
0 22 * * 6 /opt/toscience/bin/crawlReport.sh >> /opt/toscience/logs/crawlReport.log
# Create Elasticsearch und mysql - Backups in /opt/toscience/backup
0 2 * * * /opt/toscience/bin/backup-es.sh -c >> /opt/toscience/logs/backup-es.log 2>&1
30 2 * * * /opt/toscience/bin/backup-es.sh -b >> /opt/toscience/logs/backup-es.log 2>&1
0 2 * * * /opt/toscience/bin/backup-db.sh -c >> /opt/toscience/logs/backup-db.log 2>&1
30 2 * * * /opt/toscience/bin/backup-db.sh -b >> /opt/toscience/logs/backup-db.log 2>&1