pip install -r requirements.txt Failed on Ubuntu 20.04

[ange-black69]

Hello,
I am struggling to compile the omni api from source.
My goal is to have the API hosted in my own server that way it can serve easily my requests.
I have cloned the depo as usual and followed the install instruction, however, at the pip install requirements, I got the Following errors either for psycopg2 and gevent :

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for gevent

I managed to find the issues :
-> changed the version of psycopg2 from 2.x to 2.8.6
-> changed the version of gevent from 1.2.. to 21.2.2

Seems that the version not supported by python 3.8 !

Ubuntu 20.04 : kernel 5.8.0
Python 3.8

[ange-black69]

Just a quick update : Python 3 does not work with OmniApi.
I reclone from scratch and I needed to install python-dev also for making the compiling working

Also, the documentation is not very clear... I needed to manualy add the logs folder and the websocket.log and apps.log but when I start the Api, I got the following error in the websocket.log :

/usr/local/lib/python2.7/dist-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography import x509
Traceback (most recent call last):
  File "websocket.py", line 13, in <module>
    from common import *
  File "/home/mike/Documents/omniApi/omniapi/api/common.py", line 5, in <module>
    from rpcclient import *
  File "/home/mike/Documents/omniApi/omniapi/api/rpcclient.py", line 67, in <module>
    host=RPCHost()
TypeError: __init__() should return None

What I am just trying to achieve is to have the api calls working, Do I need BTC-Armory for this ?

[achamely]

Hello @ange-black69
You are correct in that Python 3 does not work,
OmniApi was originally written around python 2 and has not been migrated to for python 3 yet.
This is on the development schedule to be done but there is no timeframe available for when as of yet.
BitcoinArmory is not required unless you want access to the armory offline transaction utilities/endpoints.
Also be aware that OmniApi itself is just an interface for data stored in a database.
The Database is setup such that OmniEngine interacts with OmniCore to populate and keep the database updated.
So you need all those components working in tandem.

[ange-black69]

Hello @achamely thanks you very much for your answer !
Ok so just to let sum up, all I want to do is to have the possibility to querry the api from my own hosted server, not the one from api.omniexplorer.info because I need to send hundreads of query/seconds, and at this time the omni api kick me out (which is normal).
I have restarted from scratch yesterday trying to understand everything.
So, what is the step by step guide to host this api in a custom local server ?

At first, I have cloned the omniCore project also, and followed the insctruction to make it.
Unfortunately with ubuntu 20.04, some dependencies was removed from the depo, so I need to manually install them.
Anyway, the ./configure passed with success, and I would do the 'make' command.
Once the make is done, what is the next step ? Do I need to synch the entire BTC Blockchain to use the API ? (I think so)
Thanks you for your help,
greeting

[achamely]

@ange-black69
You can either build omnicore from source or use one of the provided releases: https://github.com/OmniLayer/omnicore/releases/tag/v0.10.0
But yes, step 1 is to get omnicore up and running and fully synced with the blockchain
Step 2 would be to setup OmniEngine and the corresponding postgres database for it to talk to
Step 3 configure OmniEngine to talk to OmniCore and (after omnicore is fully synced) start OmniEngine running to pull data from OmniCore to put in the database
Step 4, Setup OmniApi and configure it to point to the database to pull info
Step 5 once OmniEngine/Database/OmniCore are all setup, running, and 100% synced you can run your queries against it.

However if all you need to do is pull info for your own use you might be able to stop after step 1.
Omnicore has a built in rpc interface that would allow you to interact with it locally to get address balance info or tx info etc..
So depending on your use case that may be a simpler solution

[ange-black69]

Hello @achamely thanks you for your help, Yea I just need to querry my own server because for my Master Project I need to analyse every Omni Transaction that appear in the BTC BlockChain and I need to have a really good throughput like 100 to 200 transactions / seconds if possible, without overheating the current api.omnixplorer.info
So, I download the release, let it synch and then, how to configure the api to query it ?

[achamely]

@ange-black69 you wouldn't use the api, You'd just interact with omnicore's rpc commands natively: https://github.com/OmniLayer/omnicore/blob/master/src/omnicore/doc/rpc-api.md

If you're going to be analyzing Omni Transactions you'll want to use a combination of omni_listblocktransactions to get a list of tx hashes of Omni Txs in a specific block (or omni_listblockstransactions to get a range of blocks ) and then omni_gettransaction to get the detail of the txs.

Once your OC is fully synced, you can basically fire off your queries at it as fast as your machine can process them.

You can either use some bash scripting to interact with the RPC via command line omnicore-cli client or you can find a generic bitcoin rpc library for your programming language of choice that allows you to pass the commands through

[ange-black69]

ok ! that seems way more clear for me !
The only confusing thing I have now is, what is the purpose of OmniApi repo then?
How did you do to have the REST Api working for api.explorer.info/v1 etc?
What if I want to have exactly the same behavior in my custom server domain and not api.omniexplorer.info, How could I do that ?
Sorry for the questions, but it's confusing for me right now.
However, I understand a bit better the omni_* rpc commands things
The idea of the api is to call each omnicore-cli command and return the proceced json ?

EDITED :
ok I just figured it out on the documentation, there is "configuration needed". I need to create a bitcoin.conf file with rpc value in it, right ?
But where do I need to create the file ?

[achamely]

what is the purpose of OmniApi repo then?

OmniApi is a web api interface that is used to serve up OmniLayer info from a database in a more reliable fashion for multiuse Omni enabled Products (Web Wallet, Block explorer etc..)

How did you do to have the REST Api working for api.explorer.info/v1 etc?

Not sure what you are asking exactly?

What if I want to have exactly the same behavior

Then you need to setup the full stack of all the Omni Layer infrastructure components we mentioned previously (Omnicore, OmniEngine, Postgress Database, OmniApi)

idea of the api is to call each omnicore-cli command and return the proceced json ?

OmniApi acts as an in-between interface to the data in OmniCore. OmniCore itself is not suited for/nor able to handle being queried directly by hundreds/thousands of people from a website. So the data is extracted from OmniCore by OmniEngine and stored in a postgress database, then OmniApi is able to query the data from postgress database and serve it up for public use.

bitcoin.conf file with rpc value in it, right?

yes,

But where do I need to create the file ?

This is the standard bitcoin.conf file in your data directory. OmniCore will, on first launch, create the data directory and default config file (on a linux machine this will be in ~/.bitcoin/ directory) so you can update/edit this config file after its created

[ange-black69]

OK ! Thank you very very much it's really more clear for me right now !
Do you recommend to build the API with linux or can it be build easily with windows too ?

[achamely]

In theory it should be possible to build for windows as its python based however I've only ever done it on linux so i have no idea what possible issues that may present

[losh11]

@ange-black69 here's a complete guide for installation for omnilite (the litecoin version of omnicore) - just make sure to change these references and they should work. I tested with WSL2 running Ubuntu and on DO Ubuntu 20 LTS:

WSL Ubuntu 20 LTS

$ apt update
$ apt upgrade

Install OmniLite

Build or install OmniLite. Let it sync fully.

Create litecoin config file:

$ nano ~/.litecoin/litecoin.conf

Use config as shown below. Replace rpcuser and rpcpass.

testnet=0
server=1
rpcuser=losh11
rpcpassword=poopybutthole
rpcport=9332

Install omniliteEngine

omniliteEngine injects data into the postgresql db used by omni-api.

Install python2

$ sudo apt install python2

Install pip2.

$ curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
$ sudo python2 get-pip.py
$ sudo apt install libcurl4-openssl-dev libssl-dev python-dev libpq-dev

Install postgresql.

$ sudo apt install postgresql postgresql-contrib
$ sudo pg_ctlcluster 13 main start # start cluster using given output
$ sudo adduser omniliteengine
$ sudo usermod -aG sudo omniliteengine

Add new user to psql:

$ sudo - postgres
$ createuser --interactive --pwprompt # follow the prompt. y to superuser
$ createdb root # replace root with username/account name
$ exit

Add postgresql config:

$ mkdir ~/.omni
$ nano ~/.omni/sql.conf
sqluser=omniliteengine
sqlport=5432
sqlconnect=localhost
sqldatabase=omniwallet
sqlpassword=poopybutthole

Install Redis:

$ sudo apt install redis-server
$ sudo nano /etc/redis/redis.conf
supervised systemd # replace replace to use systemd
$ sudo systemctl restart redis.service

Test Redis

$ sudo service redis-server status
● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-07-29 01:25:04 UTC; 3s ago

$ redis-cli
127.0.0.1:6379> ping
PONG # expected response if everything is working
127.0.0.1:6379> exit

Clone & setup omniliteEngine.

$ git clone https://github.com/OmniLite/omniliteEngine.git
$ cd omniliteEngine
$ pip2 install python-crontab
$ pip2 install -r requirements.txt
$ cd database/scripts

# export db username/passwords etc.
# replace passwords

$ OMNIDB_DATABASE=omniwallet OMNIDB_ENGINE_USER=omniliteengine OMNIDB_ENGINE_PASSWORD=poopybutthole OMNIDB_WWW_USER=omniwww OMNIDB_WWW_PASSWORD=poopybutthole  ./db-init.sh

$ cd ../../

Test everything is working...

$ cd ~/omniliteEngine/
$ python omniEngine.py

If you see lots and lots of text going across, populating the DB, then you're ready to continue. Next setup OmniEngine as a cronjob.

$ cd ~/install/
$ python installOmniEngineCronJob.py
Adding job.
Writing job:
* * * * * python /root/omniliteEngine/omniEngine.py >> /root/omniliteEngine/logs/omniEngine.log

Install omniliteapi

$ git clone https://github.com/OmniLite/omniliteapi.git
$ cd omniliteapi
$ sudo apt-get install libpq-dev
$ pip install -r requirements.txt

Copy api/config.py.example to api/config.py and update.

$ cp api/config.py.example api/config.py
$ nano api/config.py

Copy litecoin.conf and sql.conf to omniliteapi user

$ mkdir ~/.litecoin
$ mkdir ~/.omni
$ cp /root/.litecoin/litecoin.conf ~/
$ cp /root/.omni/sql.conf ~/

Setup Nginx

$ cd ~/
$ sudo apt install nginx
$ sudo service nginx status
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-07-29 01:56:05 UTC; 37s ago

$ sudo rm /etc/nginx/sites-available/default
// loshan copy and modify from repo instead of this
$ sudo nano /etc/nginx/sites-available/default

$ apt install nginx
$ ufw allow 'Nginx Full'
$ systemctl status nginx