Metadata does not load

[stringkm]

I am running easyaccess 1.4.7 on Ubuntu 14.04, but I've had this issue in prior versions of easyaccess also. When I open easyaccess I receive the error:

$ easyaccess
Connecting to DB ** dessci ** ...
Loading metadata into cache...
There was an error when refreshing the metadata

Also, when I type "refresh_metadata_cache", I get the same error:

DESSCI ~> refresh_metadata_cache
Loading metadata into cache...
There was an error when refreshing the metadata

Without the metadata, I am unable to use basic functions such as "find_tables" which gives:

DESSCI ~> find_tables %y3a1%
| ☆ | Ctrl-C to abort;
<class 'ValueError'>
array size too large

and "describe_table" which gives:

DESSCI ~> describe_table y3a2_coadd_object_summary
Table not found.

I tried uninstalling using "conda remove easyaccess", then reinstalling using conda, but the error did not change. Is there a way to force easyaccess to download a fresh copy of the metadata or download it manually?

[mgckind]

I think this is also related to #152 , @madamow

[madamow]

@stringkm could you open easy access and run the 3 following queries? Let me know if they work, and if they not -copy/paste an error they return.

select table_name from DES_ADMIN.CACHE_TABLES union select table_name from user_tables;
select distinct username from dba_users order by username;
SELECT column_name from DES_ADMIN.CACHE_COLUMNS;

[stringkm]

Here are the results:

DESSCI ~> select table_name from DES_ADMIN.CACHE_TABLES union select table_name from user_tables;

| ☆            |  Ctrl-C to abort; 

<class 'ValueError'>
array size too large

DESSCI ~> select distinct username from dba_users order by username;

| ☆            |  Ctrl-C to abort; 

<class 'ValueError'>
array size too large

The third query was successful.

[madamow]

Please try this:

1. change name of your .desservices.ini file
2. set up a new Conda environment using python 2.7:
   conda create -n eatest python=2.7 anaconda
3. close and open your terminal
4. activate new environment
   conda activate eatest
5. install easy access with Conda using new environment (line in terminal should start with (eatest)
   conda install easyaccess==1.4.7 -c mgckind -c anaconda
6. run easy access, log in and check those three queries again and let me know if this makes any difference

[stringkm]

Unfortunately, even after reinstalling with a new environment, the results are similar, but the error codes are slightly different (maybe due to using python 2 instead of 3?).

(eatest) :~$ easyaccess

DESSCI ~> select table_name from DES_ADMIN.CACHE_TABLES union select table_name from user_tables;

<type 'exceptions.ValueError'>
array size too large

DESSCI ~> select distinct username from dba_users order by username;

<type 'exceptions.ValueError'>
array size too large

The last query was successful.

[mgckind]

Thanks @stringkm for the help in figuring this ou! t, I believe both are the same errors (which have appeared before)

[madamow]

Ok, I will try to recreate this error on Ubuntu. Thanks.

[madamow]

One more thing. Could you paste me here the result from uname -a please?

[stringkm]

Linux antec-Z87-D3HP 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[madamow]

I setup docker with your version of Ubuntu and easyaccess works normally.
Two more things that we can check:
content of ~/.easyaccess/config.ini
and output of conda activate eatest ; conda export env
If there will be no differences here - I am out of ideas.

[mgckind]

@stringkm, @madamow this is puzzling, as I haven't been able to reproduce the problem either :( andis related also to #152
can please you try using cx_oracle directly, to check whether the queries return successfully?
By opening python and then

import easyaccess as ea
import cx_Oracle as cx
print(cx.__version__)
con = ea.connect()
con2 = cx.connect(con.user, con.password, dsn=con.dsn)
cursor = con2.cursor()
query = 'select table_name from DES_ADMIN.CACHE_TABLES union select table_name from user_tables'
con.query_and_print(query) # this is easyaccess
cursor.execute(query).fetchall() # this is cx_Orace

[stringkm]

Sorry I took a day to do that. I had a final exam this afternoon. Here's the output from those commands. The cx_oracle command at the end was successful, but I did not include the output so I wouldn't compromise any of the private table names.

[mgckind]

Thanks!
Interesting, so it seems that converting from Oracle output to pandas/numpy seems to be the issue,
maybe trying to allocate the data to actually get the oracle data is where it fails, @madamow , I don't have a good explanation for this but it might be a combination of system/pandas/cx_Oracle version ... we'll keep digging, Thanks @stringkm

[stringkm]

No problem. In case you need it, my pandas version is 0.24.0 and my numpy version is 1.14.3.

[madamow]

This is why I wanted to see conda activate eatest ; conda export env output. To be sure that the combination of packages is the same. @stringkm could you run it and paste the output here?

[stringkm]

(base) katelyn@antec-Z87-D3HP:~$ conda activate eatest ; conda export env

CommandNotFoundError: No command 'conda export'.

(eatest) katelyn@antec-Z87-D3HP:$ conda activate eatest
(eatest) katelyn@antec-Z87-D3HP:$ conda export env

CommandNotFoundError: No command 'conda export'.

[madamow]

sorry, conda env export

[stringkm]

(eatest) katelyn@antec-Z87-D3HP:~$ conda env export
WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.compat module is deprecated and will be removed in a future release.
name: eatest
channels:

• mgckind
• anaconda
• defaults
  dependencies:
• alabaster=0.7.12=py27_0
• anaconda=custom=py27h4a00acb_0
• anaconda-client=1.7.2=py27_0
• anaconda-project=0.8.2=py27_0
• asn1crypto=0.24.0=py27_0
• astroid=1.6.5=py27_0
• astropy=2.0.9=py27hdd07704_0
• atomicwrites=1.3.0=py27_1
• attrs=19.1.0=py27_1
• babel=2.6.0=py27_0
• backports=1.0=py27_1
• backports.functools_lru_cache=1.5=py27_1
• backports.os=0.1.1=py27_0
• backports.shutil_get_terminal_size=1.0.0=py27_2
• backports_abc=0.5=py27h7b3c97b_0
• beautifulsoup4=4.7.1=py27_1
• bitarray=0.8.3=py27h14c3975_0
• bkcharts=0.2=py27h241ae91_0
• blas=1.0=mkl
• bleach=3.1.0=py27_0
• blosc=1.15.0=hd408876_0
• bokeh=1.0.4=py27_0
• boto=2.49.0=py27_0
• bottleneck=1.2.1=py27h035aef0_1
• bzip2=1.0.6=h14c3975_5
• ca-certificates=2019.1.23=0
• cairo=1.14.12=h8948797_3
• cdecimal=2.3=py27h14c3975_3
• certifi=2019.3.9=py27_0
• cffi=1.12.2=py27h2e261b9_1
• chardet=3.0.4=py27_1
• click=7.0=py27_0
• cloudpickle=0.8.0=py27_0
• clyent=1.2.2=py27_1
• colorama=0.4.1=py27_0
• configparser=3.7.3=py27_1
• contextlib2=0.5.5=py27hbf4c468_0
• cryptography=2.6.1=py27h1ba5d50_0
• curl=7.64.0=hbc83047_2
• cx_oracle=5.3=py27_0
• cycler=0.10.0=py27hc7354d3_0
• cython=0.29.6=py27he6710b0_0
• cytoolz=0.9.0.1=py27h14c3975_1
• dask=1.1.4=py27_1
• dask-core=1.1.4=py27_1
• dbus=1.13.6=h746ee38_0
• decorator=4.4.0=py27_1
• defusedxml=0.5.0=py27_1
• distributed=1.26.0=py27_1
• docutils=0.14=py27hae222c1_0
• easyaccess=1.4.7=py27_0
• entrypoints=0.3=py27_0
• enum34=1.1.6=py27_1
• et_xmlfile=1.0.1=py27h75840f5_0
• expat=2.2.6=he6710b0_0
• fastcache=1.0.2=py27h14c3975_2
• filelock=3.0.10=py27_0
• fitsio=0.9.11=py27_1
• flask=1.0.2=py27_1
• fontconfig=2.13.0=h9420a91_0
• freetype=2.9.1=h8a8886c_1
• fribidi=1.0.5=h7b6447c_0
• funcsigs=1.0.2=py27h83f16ab_0
• functools32=3.2.3.2=py27_1
• future=0.17.1=py27_0
• futures=3.2.0=py27_0
• get_terminal_size=1.0.0=haa9412d_0
• gevent=1.4.0=py27h7b6447c_0
• glib=2.56.2=hd408876_0
• glibc=2.12.2=3
• glob2=0.6=py27_1
• gmp=6.1.2=h6c8ec71_1
• gmpy2=2.0.8=py27h10f8cd9_2
• graphite2=1.3.13=h23475e2_0
• greenlet=0.4.15=py27h7b6447c_0
• grin=1.2.1=py27_4
• gst-plugins-base=1.14.0=hbbd80ab_1
• gstreamer=1.14.0=hb453b48_1
• h5py=2.9.0=py27h7918eee_0
• harfbuzz=1.8.8=hffaf4a1_0
• hdf5=1.10.4=hb1b8bf9_0
• heapdict=1.0.0=py27_2
• html5lib=1.0.1=py27_0
• icu=58.2=h9c2bf20_1
• idna=2.8=py27_0
• imageio=2.5.0=py27_0
• imagesize=1.1.0=py27_0
• importlib_metadata=0.8=py27_0
• intel-openmp=2019.3=199
• ipaddress=1.0.22=py27_0
• ipykernel=4.10.0=py27_0
• ipython=5.8.0=py27_0
• ipython_genutils=0.2.0=py27h89fb69b_0
• ipywidgets=7.4.2=py27_0
• isort=4.3.16=py27_0
• itsdangerous=1.1.0=py27_0
• jbig=2.1=hdba287a_0
• jdcal=1.4=py27_0
• jedi=0.13.3=py27_0
• jinja2=2.10=py27_0
• jpeg=9b=h024ee3a_2
• jsonschema=3.0.1=py27_0
• jupyter=1.0.0=py27_7
• jupyter_client=5.2.4=py27_0
• jupyter_console=5.2.0=py27_1
• jupyter_core=4.4.0=py27_0
• jupyterlab=0.33.11=py27_0
• jupyterlab_launcher=0.11.2=py27h28b3542_0
• kiwisolver=1.0.1=py27hf484d3e_0
• krb5=1.16.1=h173b8e3_7
• lazy-object-proxy=1.3.1=py27h14c3975_2
• libarchive=3.3.3=h5d8350f_5
• libcurl=7.64.0=h20c2e04_2
• libedit=3.1.20181209=hc058e9b_0
• libffi=3.2.1=hd88cf55_4
• libgcc-ng=8.2.0=hdf63c60_1
• libgfortran-ng=7.3.0=hdf63c60_0
• liblief=0.9.0=h7725739_2
• libpng=1.6.36=hbc83047_0
• libsodium=1.0.16=h1bed415_0
• libssh2=1.8.0=h1ba5d50_4
• libstdcxx-ng=8.2.0=hdf63c60_1
• libtiff=4.0.10=h2733197_2
• libtool=2.4.6=h7b6447c_5
• libuuid=1.0.3=h1bed415_2
• libxcb=1.13=h1bed415_1
• libxml2=2.9.9=he19cac6_0
• libxslt=1.1.33=h7d1a2b0_0
• linecache2=1.0.0=py27_0
• llvmlite=0.28.0=py27hd408876_0
• locket=0.2.0=py27h73929a2_1
• lxml=4.3.2=py27hefd8a0e_0
• lz4-c=1.8.1.2=h14c3975_0
• lzo=2.10=h49e0be7_2
• markupsafe=1.1.1=py27h7b6447c_0
• matplotlib=2.2.3=py27hb69df0a_0
• mccabe=0.6.1=py27_1
• mistune=0.8.4=py27h7b6447c_0
• mkl=2019.3=199
• mkl-service=1.1.2=py27he904b0f_5
• mkl_fft=1.0.10=py27ha843d7b_0
• mkl_random=1.0.2=py27hd81dba3_0
• more-itertools=5.0.0=py27_0
• mpc=1.1.0=h10f8cd9_1
• mpfr=4.0.1=hdf1c602_3
• mpmath=1.1.0=py27_0
• msgpack-python=0.6.1=py27hfd86e86_1
• multipledispatch=0.6.0=py27_0
• nbconvert=5.4.1=py27_3
• nbformat=4.4.0=py27hed7f2b2_0
• ncurses=6.1=he6710b0_1
• networkx=2.2=py27_1
• nltk=3.4=py27_1
• nose=1.3.7=py27_2
• notebook=5.7.8=py27_0
• numba=0.43.1=py27h962f231_0
• numexpr=2.6.9=py27h9e4a6bb_0
• numpy=1.16.2=py27h7e9f1db_0
• numpy-base=1.16.2=py27hde5b4d6_0
• numpydoc=0.8.0=py27_0
• olefile=0.46=py27_0
• openpyxl=2.6.1=py27_1
• openssl=1.1.1=h7b6447c_0
• oracle-instantclient=11.2.0.4.0=0
• packaging=19.0=py27_0
• pandas=0.24.2=py27he6710b0_0
• pandoc=2.2.3.2=0
• pandocfilters=1.4.2=py27_1
• pango=1.42.4=h049681c_0
• parso=0.3.4=py27_0
• partd=0.3.10=py27_1
• patchelf=0.9=he6710b0_3
• path.py=11.5.0=py27_0
• pathlib2=2.3.3=py27_0
• patsy=0.5.1=py27_0
• pcre=8.43=he6710b0_0
• pep8=1.7.1=py27_0
• pexpect=4.6.0=py27_0
• pickleshare=0.7.5=py27_0
• pillow=5.4.1=py27h34e0f95_0
• pip=19.0.3=py27_0
• pixman=0.38.0=h7b6447c_0
• pkginfo=1.5.0.1=py27_0
• pluggy=0.9.0=py27_0
• ply=3.11=py27_0
• prometheus_client=0.6.0=py27_0
• prompt_toolkit=1.0.15=py27h1b593e1_0
• psutil=5.6.1=py27h7b6447c_0
• ptyprocess=0.6.0=py27_0
• py=1.8.0=py27_0
• py-lief=0.9.0=py27h7725739_2
• pycairo=1.18.0=py27h2a1e443_0
• pycodestyle=2.5.0=py27_0
• pycosat=0.6.3=py27h14c3975_0
• pycparser=2.19=py27_0
• pycrypto=2.6.1=py27h14c3975_9
• pycurl=7.43.0.2=py27h1ba5d50_0
• pyflakes=2.1.1=py27_0
• pygments=2.3.1=py27_0
• pylint=1.9.2=py27_0
• pyodbc=4.0.26=py27he6710b0_0
• pyopenssl=19.0.0=py27_0
• pyparsing=2.3.1=py27_0
• pyqt=5.9.2=py27h05f1152_2
• pyrsistent=0.14.11=py27h7b6447c_0
• pysocks=1.6.8=py27_0
• pytables=3.5.1=py27h71ec239_0
• pytest=4.3.1=py27_0
• python=2.7.16=h9bab390_0
• python-dateutil=2.8.0=py27_0
• python-libarchive-c=2.8=py27_6
• pytz=2018.9=py27_0
• pywavelets=1.0.2=py27hdd07704_0
• pyyaml=5.1=py27h7b6447c_0
• pyzmq=18.0.0=py27he6710b0_0
• qt=5.9.7=h5867ecd_1
• qtawesome=0.5.7=py27_1
• qtconsole=4.4.3=py27_0
• qtpy=1.7.0=py27_1
• readline=7.0=h7b6447c_5
• requests=2.21.0=py27_0
• rope=0.12.0=py27_0
• ruamel_yaml=0.15.46=py27h14c3975_0
• scandir=1.10.0=py27h7b6447c_0
• scikit-image=0.14.2=py27he6710b0_0
• scikit-learn=0.20.3=py27hd81dba3_0
• scipy=1.2.1=py27h7c811a0_0
• seaborn=0.9.0=py27_0
• send2trash=1.5.0=py27_0
• setuptools=40.8.0=py27_0
• simplegeneric=0.8.1=py27_2
• singledispatch=3.4.0.3=py27h9bcb476_0
• sip=4.19.8=py27hf484d3e_0
• six=1.12.0=py27_0
• snappy=1.1.7=hbae5bb6_3
• snowballstemmer=1.2.1=py27h44e2768_0
• sortedcollections=1.1.2=py27_0
• sortedcontainers=2.1.0=py27_0
• soupsieve=1.8=py27_0
• sphinx=1.8.5=py27_0
• sphinxcontrib=1.0=py27_1
• sphinxcontrib-websupport=1.1.0=py27_1
• spyder=3.3.3=py27_0
• spyder-kernels=0.4.2=py27_0
• sqlalchemy=1.3.1=py27h7b6447c_0
• sqlite=3.27.2=h7b6447c_0
• ssl_match_hostname=3.7.0.1=py27_0
• statsmodels=0.9.0=py27h035aef0_0
• subprocess32=3.5.3=py27h7b6447c_0
• sympy=1.3=py27_0
• tblib=1.3.2=py27h51fe5ba_0
• termcolor=1.1.0=py27_0
• terminado=0.8.1=py27_1
• testpath=0.4.2=py27_0
• tk=8.6.8=hbc83047_0
• toolz=0.9.0=py27_0
• tornado=5.1.1=py27h7b6447c_0
• tqdm=4.31.1=py27_1
• traceback2=1.4.0=py27_0
• traitlets=4.3.2=py27hd6ce930_0
• typing=3.6.6=py27_0
• unicodecsv=0.14.1=py27h5062da9_0
• unittest2=1.1.0=py27_0
• unixodbc=2.3.7=h14c3975_0
• urllib3=1.24.1=py27_0
• wcwidth=0.1.7=py27h9e3e1ab_0
• webencodings=0.5.1=py27_1
• werkzeug=0.14.1=py27_0
• wheel=0.33.1=py27_0
• widgetsnbextension=3.4.2=py27_0
• wrapt=1.11.1=py27h7b6447c_0
• wurlitzer=1.0.2=py27_0
• xlrd=1.2.0=py27_0
• xlsxwriter=1.1.5=py27_0
• xlwt=1.3.0=py27h3d85d97_0
• xz=5.2.4=h14c3975_4
• yaml=0.1.7=had09818_2
• zeromq=4.3.1=he6710b0_3
• zict=0.1.4=py27_0
• zipp=0.3.3=py27_1
• zlib=1.2.11=h7b6447c_3
• zstd=1.3.7=h0b5b093_0
  prefix: /home/katelyn/anaconda3/envs/eatest

[madamow]

...and content of ~/.easyaccess/config.ini

[stringkm]

[easyaccess]
database = dessci
editor = nano
prefetch = 30000000
histcache = 5000
timeout = 900
nullvalue = -9999
outfile_max_mb = 10000
autocommit = yes
trim_whitespace = yes
desdm_coldefs = yes
compression = no

[display]
color_terminal = yes
loading_bar = yes
max_rows = 1500
width = 1000
max_columns = 50
max_colwidth = 500

[madamow]

Ok, I (think I) know what is wrong.
Change the prefetch parameter. To 30 000, and if it doesn't work, use smaller number.

[stringkm]

Yes, I think that was the culprit. With prefetch set to 30000, basic function calls like find_tables and describe_table seem to work.

[madamow]

With those settings you tried to use more memory than you had available. And that was the problem.
Good luck with your finals!

[stringkm]

Thanks!

[mgckind]

Thanks both @madamow and @stringkm, definitely a new pair of eyes helped,as i said in the other issue, i completely ruled out this solution as it never occurred to me it was returning that many rows, still is weird (for small tables) it complains, there isba leak somewhere but this is good news anyways. Thanks