webapi refined, need to finish conversion

Author: twhughes

.pylintrc

@@ -10,4 +10,4 @@ max-line-length=100
 
 [pre-commit-hook]
 command=custom_pylint
-disable=pointless-string-statement, too-many-ancestors, too-few-public-methods, fixme, logging-not-lazy
+disable=pointless-string-statement, too-many-ancestors, too-few-public-methods, fixme, logging-not-lazy, logging-fstring-interpolation

README.md

@@ -157,9 +157,9 @@ https://www.atlassian.com/git/tutorials/merging-vs-rebasing
 	- [x] **Refactor some webapi internals.**
 	- [ ] **Add conversions for rest of objects.**
 	- [x] **Containers (job batch).**
-	- [ ] **Better handling for runtime status using rich.**
+	- [x] **Better handling for runtime status using rich.**
 	- [ ] **Add example notebooks and make consistent.**
-	- [ ] **Comments / documentations**
+	- [x] **Comments / documentations**
 - [ ] Get webAPI working without conversion.
 	- [ ] Use native `Simulation.export()` or `Simulation.json()` for `upload()`.
 	- [ ] Put the tidy3d log inside SimulationData on postprocess.

docs/StartHere.py

@@ -1,69 +1,74 @@
 #!/usr/bin/env python
 # coding: utf-8
 
+get_ipython().run_line_magic("load_ext", "autoreload")
+get_ipython().run_line_magic("autoreload", "2")
+
+import matplotlib.pylab as plt
+
 import sys
 
 sys.path.append("..")
 
 import tidy3d as td
+import tidy3d.web as web
+
 
 # set up parameters of simulation
-dl = 0.1
+dl = 0.01
 pml = td.PMLLayer(profile="standard", num_layers=10)
 sim_size = [4, 4, 4]
 freq0 = 3e14
 fwidth = 1e13
-run_time = 2 / fwidth
+run_time = 10 / fwidth
 
 # create structure
 dielectric = td.nk_to_medium(n=2, k=0, freq=freq0)
 square = td.Structure(geometry=td.Box(center=[0, 0, 0], size=[1.5, 1.5, 1.5]), medium=dielectric)
 
 # create source
-source = td.PlaneWave(
-    center=(0, -1.5, 0),
-    size=(td.inf, 0, td.inf),
+source = td.VolumeSource(
+    center=(-1.5, 0, 0),
+    size=(0, 0.4, 0.4),
     source_time=td.GaussianPulse(freq0=freq0, fwidth=fwidth),
     polarization="Jx",
-    direction="+",
 )
 
 # create monitor
-monitor = td.FieldMonitor(
-    fields=["Ex", "Hy"], center=(0, 0, 0), size=(td.inf, 0, td.inf), freqs=[freq0]
-)
+monitor = td.FieldMonitor(fields=["Ex", "Hy"], center=(0, 0, 0), size=(4, 4, 0), freqs=[freq0])
 
 # Initialize simulation
 sim = td.Simulation(
     size=sim_size,
     grid_size=(dl, dl, dl),
     structures=[square],
-    sources={"plane_wave": source},
-    monitors={"field_monitor": monitor},
+    sources={"source": source},
+    monitors={"monitor": monitor},
     run_time=run_time,
     pml_layers=(pml, pml, pml),
 )
 
+task_id = web.upload(sim, task_name="quickstart")
+web.start(task_id)
+web.monitor(task_id)
 
-# # run simulation
-# project = web.new_project(sim.export(), task_name='quickstart')
-# web.monitor_project(project['taskId'])
 
-# # download results
-# web.download_results(project['taskId'], target_folder='out/')
-# sim.load_results('out/monitor_data.hdf5')
+web.download(task_id, simulation=sim, path="data/sim_data.hdf5")
+
+
+sim_data = web.load(task_id, simulation=sim, path="data/sim_data.hdf5")
+ax = (
+    sim_data["monitor"]
+    .Ex.isel(f=0, z=0)
+    .imag.plot.pcolormesh(x="x", y="y", vmin=-5e-13, vmax=5e-13, cmap="RdBu")
+)
+
+plt.show()
 
-# # get field data as a numpy array
-# E = sim.data(freq_mnt)["E"]
 
-# # plot results
-# clim = (1, 3)
-# fig, (ax_top, ax_bot) = plt.subplots(2, 2, figsize=(9, 7), tight_layout=True)
+from pprint import pprint as print
 
-# sim.viz_eps_2D(normal='y', ax=ax_top[0], cbar=True, clim=clim, monitor_alpha=0)
-# im_re = sim.viz_field_2D(freq_mnt, ax=ax_top[1], cbar=True, comp='y', val='re')
+print(sim_data.task_info)
 
-# im_ab = sim.viz_field_2D(freq_mnt, ax=ax_bot[0], cbar=True, comp='y', val='abs')
-# im_int = sim.viz_field_2D(freq_mnt, ax=ax_bot[1], cbar=True, comp='y', val='int')
 
-# plt.show()
+sim_data.log

docs/api.rst

@@ -81,6 +81,8 @@ Simulation Output Data
 Submitting Simulations
 ======================
 
+.. currentmodule:: tidy3d
+
 Web API
 -------
 
@@ -93,7 +95,7 @@ Web API
    web.run
    web.monitor
    web.download
-   web.load_results
+   web.load_data
    web.delete
 
 Job Interface
@@ -103,14 +105,6 @@ Job Interface
    :toctree: _autosummary/
 
    web.Job
-   web.Job.upload
-   web.Job.get_info
-   web.Job.get_run_info
-   web.Job.run
-   web.Job.monitor
-   web.Job.download
-   web.Job.load_results
-   web.Job.delete   
 
 Batch Processing
 ----------------
@@ -119,14 +113,6 @@ Batch Processing
    :toctree: _autosummary/
 
    web.Batch
-   web.Batch.upload
-   web.Batch.get_info
-   web.Batch.get_run_info
-   web.Batch.run
-   web.Batch.monitor
-   web.Batch.download
-   web.Batch.load_results
-   web.Batch.delete
 
 Info Containers
 ---------------

docs/howdoi.rst

@@ -97,3 +97,29 @@ Plot Data
      - ``mon_data.plot()`` if data is 2D, otherwise one can use ``mon_data.plot.pcolormesh(x='x', y='y')`` to specify the ``x`` and ``y`` coordinates explicitly.
    * - Plot the simulation structure on top of my field plot?
      - ``sim_data.simulation.plot_structures_eps(z=0.0, ax=ax, alpha=.5, cbar=False, lw=0)`` will plot the ``z=0`` cross-section, where ``ax`` is the axis where your fields are plotted on.
+
+Submit Jobs to Server
+---------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 40 60
+
+   * - How do I...
+     - Solution
+   * - Submit my simulations to run on Flexcompute's servers?
+     - The simplest way is to create a :class:`Job` object from your :class:`Simulation` object and call the ``.run()`` method, which does all steps.  For example:  ``job = Job(simulation=simulation, task_name='my_sim');  job.run()``.
+   * - Upload a job to the web without running it so I can inspect it first?
+     - Once you've created a :class:`Job`, you can upload it to our servers with ``job.upload()`` and it will not run until you excplicitly tell it to with ``job.start()``.
+   * - Monitor the progress of a simulation?
+     - ``web.monitor(task_id)``, ``Job.monitor()``, or ``Batch.monitor()`` will display the progress of your simulation(s).
+   * - Load the results of a simulation?
+     - ``sim_data = job.load_results(path)`` will download the results to ``path`` and load them as :class:`SimulationData` object.
+   * - See information about my :class:`Job`, such as how many credits it will take?
+     - After uploading your job with ``job.upload()`` you can get a host of information about it through ``task_info = job.get_info()``.
+   * - Submit multiple simulations?
+     - The :class:`Batch` interface was created to manage multiple :class:`Job` instances and gives a similar interface with large number of jobs in mind.
+   * - Loop through :class:`Batch` data without loading all of the data into memory?
+     - ``for task_name, sim_data in batch.items():`` will ``yield`` the :class:`SimulationData` for each :class:`Job` in the batch one by one, so you can perform your postprocessing in the loop body without loading each of the simulations' data into memory at once.
+   * - Save or load a :class:`Job` or :class:`Batch` so I can work with it later?
+     - Like most other tidy3d objects, :class:`Job` and :class:`Batch` instances have ``.export(path)`` and ``.load()`` methods that will export and load thier metadata as .json files.  This is especially useful for loading batches for analysis long after they have run.