Skip to content
This repository was archived by the owner on Nov 27, 2019. It is now read-only.

Frequently asked questions

Jump to bottom
Steven Maude edited this page Mar 26, 2015 · 10 revisions

Does the code carry on running if I close my browser window?

Yes. The run is not attached to your browser, it carries on in the background on the server until it finishes, or you stop it. Only one copy of your script can be running at a time. If you happen to log in while a scheduled run is happening, you'll be able to see it taking place.

How do I see the output of a scheduled run?

Simply go to the "Code in your browser" tool and the output of the last time the script was run is in the console at the bottom. This applies whether it was run by hand or scheduled.

Can my script run forever?

No, they're limited to an hour. You shouldn't need that long though! Write your scripts so that when they restart they get more data, rather than doing work they've already done. Then schedule them to run again every day or hour.

Why does the start of my script's output sometimes not appear?

Only the last 10,000 lines are shown. This is to speed up how quickly it is shown. You can find the full log files in the file logs/out if you SSH into the "Code in your browser" tool.

How are errors reported?

When your script exits without an error (i.e. exit code 0), the "Code in your browser" tool sets the ScraperWiki dataset status to "ok". If it exits with an error (if there's an exception, or otherwise a non-zero exit code) then it sets the status to "error". This status is shown on your data hub home page on the dataset.

How do I edit the code using vim or Emacs?

First, SSH into the "Code in your browser" tool. You can do that from the cog settings menu for the tool. If this is the first time you've SSHd into ScraperWiki, follow the instructions to set up your keys.

Then you can use normal shell commands to browse round the tool. Your code is in the code directory in the file scraper. You can use any tool to edit the file, remember to reload in the browser editor afterwards though!

For more information about using SSH see the Develop on the box section of the Developer Docs.

Can I get to an old version of my script?

Your code is all saved in a local git repository, automatically committed to each time you ran it. You'll need to SSH in and use the command line to access it, or ask ScraperWiki support for help.

$ cd code
$ git log

You can, if you know how to use git, add a remote and push that repository to github or similar.

How do I install Python packages?

Easiest way is:

  1. SSH into the box.
  2. Install the package locally: pip install --user <packagename>

How do I install R packages?

  1. SSH into the box.
  2. Run R by entering the command R
  3. In R, enter the command: install.packages('package_name_that_you_want_to_install)` substituting in the name of the package you want to install.
  4. R will prompt you with the warning: 'lib = "/usr/local/lib/R/site-library"' is not writable and ask you if you want to use a personal library. Enter y to use a personal library.
  5. Enter y when R asks you if you want to create a personal library to install packages into.
  6. Choose a CRAN mirror: it doesn't matter too much which you pick. The package will then install.

Alternatively, before running R, you can SSH into the box, create a .Renviron file in /home that contains the line R_LIBS=/home/r_packages and just do steps 2, 3 and 6 above. (This way, you've already specified that you want to use a personal library.)

Where can I ask more questions?

Best place is on the ScraperWiki Google Group.

Clone this wiki locally