Jackz merge ODLA POPART code from ipu_stable_sdk2.3.1 to master (#898)

* update config to poplar sdk 2.2

* update CMakeList to 2.2

* add new error code macro

(cherry picked from commit 00ac92e)

* translate poplar exception to odla_status error

* fix lint error

* fix lint error

* translate poplar exception to odla_status error

* [IPU] Move to SDK 2.2.2+711-26aba6cf16

* cherry-pick master code to branch SDK2.2.2 (#559)

* Add Custom Op for Yolov3 Post Process (#512)

* add custom op for yolov3

* reset submodule onnx

* reset tensorrt

* delete build

* merge odla_ops_nn

* modify for passing link-check

Co-authored-by: gcuser <jackz@graphcore.ai>
(cherry picked from commit 5847cd3)

* ODLA popART pipeline function (#522)

* First runnable with single thread & test context

* mnist runnable demot to test the pipeline

* multi thread put the data to the session run

* simple bash to compile and run test

* An example of how to use the callback in pipeline

* multi threads using local Ctx

* Can run with pipeline setting in onnx file

* Refactored and add no pipeline multi thread

* Move codes to the odla_pipeline.h .cc

* Make single empty/zero data, and delete context for empty data after get result

* Add mutex to serialization the compute requests

* Merge the changes for attention mask & prevous changes

* test codes for time

* Chage the CMakeList to make the pipeline.cc and new custom op compiled

* Successfully run on 24L with attention mask custom OP

* custom op attention_mask test code

* And name scope to the each node in model

* Try throghput test with MLPerf model

* only set AMP on feed forward matmul

* Run the online pipeling with config hard coded to the config read class

* Compile with SDK 2.2 with pipeline online setting

* Add config file for pipeline stage setting

* Run pipeline with similar performance of popart

* change some names & make AMP all 0.445

* Add amp parameter in config file

* Detach device and clear session when DestroyComputation

* Make the batch_per_step take effect on execution mode SEQUENCE to pass enough size of data

* Add the new lock free queue and logging

* Fix bug on empty data visit counter

* delete the empty context

* add some pipeline sync

* Make thread sleep for 5 ms when no task in the queue

* change the size() of LockFreeQueue to tail-wait

* [CI] make the call by main can work with npz files

* Move the computation init to create context

* Add common functions to common.h and common.cc

* move the compuation init out

* Move common functions to the test foler

* Test the config of ODLA popART and make no configuration act as before

* Add tests for call the model.cc

* Add FP32 to save as result

* Some changes on LockFreeQueue and tests

* Fix the rsqrt wrong problem, and remove std cout&cerr to avoid crash

* fix the accuracy problem of large bps

* Add thread check for context & computation holding to avoid conflicts

* Add the batch tools to help on the test to generate model, build and run

* Decreasing the empty data put

* temporary commit to migrate crashed system

* set pipeline information on fly
change the mixed style of class member
add debug setting and default to false to make the opts set by api
remove the old pipeline set api

* Fixed the mixed code style and removed redundant codes

* Remove the function test codes of the odla_popart

* remove some redundant codes and files

* Changed the CACHE STRING to CACHE PATH

* move ENGINE_CACHE_PATH to odla_popart.cc

* format the codes with clang-format-9 -i command

* Move json.hpp to third party

* Set virtualgraph for model not using pipeline in set_session_opts

* Add virtual graph attribute when _odla_computation constructed

* Check the shape before extends it with batches_per_step

Co-authored-by: gcuser <gcuser@alibaba-inc.com>
(cherry picked from commit 6095bdf)

* fix on default configuration & computation destroyment

(cherry picked from commit 40b9fc8)

* definitions for static variables

(cherry picked from commit 18e0e83)

* disable test case test_constant_popart.cc

Co-authored-by: Zars19 <1036473307@qq.com>
Co-authored-by: jackzipu <74961298+jackzipu@users.noreply.github.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* Combine config cache (#659) (#663)

* fix poplar sdk path symbol

* deserialize config in catch file

* Add error handling codes

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@alibaba-inc.com>
Co-authored-by: gcuser <gcuser@alibaba-inc.com>
(cherry picked from commit 0b006c0)

Co-authored-by: yanwei-gr <64010848+yanwei-gr@users.noreply.github.com>

* close floating point check in popart (#665)

* close floating point check in popart

* create pipeline resource when computation created. not do it at the time of set the cache computation item

fix pixel bert detect application runtime error

* add weiming build script fix

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>

* cherry pick the CreateExecutable updates (#671)

* close floating point check in popart

* create pipeline resource when computation created. not do it at the time of set the cache computation item

fix pixel bert detect application runtime error

* add weiming build script fix

* add inplementation of CreateExecutable

move complie_and_run to createExecute, and more error handle logic

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>

* catch potential exception and handle (#677)

* close floating point check in popart

* create pipeline resource when computation created. not do it at the time of set the cache computation item

fix pixel bert detect application runtime error

* add weiming build script fix

* add inplementation of CreateExecutable

move complie_and_run to createExecute, and more error handle logic

* add potential exception handle & open session_option to optimize

* add more exception handling

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* add bool to make sure a compuation can't be executed when used to compile executable

* improve load config logic

* updat noticee strings

* update load_config logic bug

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>

* More exception handler & CreateExecutable return fail if computation init failed. (#685)

* close floating point check in popart

* create pipeline resource when computation created. not do it at the time of set the cache computation item

fix pixel bert detect application runtime error

* add weiming build script fix

* add inplementation of CreateExecutable

move complie_and_run to createExecute, and more error handle logic

* add potential exception handle & open session_option to optimize

* add more exception handling

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* add bool to make sure a compuation can't be executed when used to compile executable

* improve load config logic

* updat noticee strings

* update load_config logic bug

* add more exception handler

* stop compiling and reture ODLA_FAILURE when Computation init failed

* fix lint error

* finx lint error

* add more exception handler

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>

* reset popart_config when destroyComputation & reset fstream pointer when loading (#687)

* close floating point check in popart

* create pipeline resource when computation created. not do it at the time of set the cache computation item

fix pixel bert detect application runtime error

* add weiming build script fix

* add inplementation of CreateExecutable

move complie_and_run to createExecute, and more error handle logic

* add potential exception handle & open session_option to optimize

* add more exception handling

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* add bool to make sure a compuation can't be executed when used to compile executable

* improve load config logic

* updat noticee strings

* update load_config logic bug

* add more exception handler

* stop compiling and reture ODLA_FAILURE when Computation init failed

add more exception handler

* Check the sdk version of the cache

* Format the codes to pass lint checking

* reset popart_config when destroyComputation & reset fstream pointer when loading

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* reset other important popart_config values when destroy computation (#689)

* close floating point check in popart

* create pipeline resource when computation created. not do it at the time of set the cache computation item

fix pixel bert detect application runtime error

* add weiming build script fix

* add inplementation of CreateExecutable

move complie_and_run to createExecute, and more error handle logic

* add potential exception handle & open session_option to optimize

* add more exception handling

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* add bool to make sure a compuation can't be executed when used to compile executable

* improve load config logic

* updat noticee strings

* update load_config logic bug

* add more exception handler

* stop compiling and reture ODLA_FAILURE when Computation init failed

add more exception handler

* Check the sdk version of the cache

* Format the codes to pass lint checking

* reset popart_config when destroyComputation & reset fstream pointer when loading

* reset other necessary popart config value

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* add return value for comp::init & add trunc when open cache_fs & do clear after close cache_fs (#691)

* add potential exception handle & open session_option to optimize

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* improve load config logic

* Check the sdk version of the cache

* Format the codes to pass lint checking

* reset other necessary popart config value

* add return value in computation::init()

* clear cache_fs before write new data

* clear fstream after close

* fix lint error

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* update error message notice & remove useless } (#693)

* add potential exception handle & open session_option to optimize

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* improve load config logic

* Check the sdk version of the cache

* Format the codes to pass lint checking

* reset other necessary popart config value

* add return value in computation::init()

* clear cache_fs before write new data

* clear fstream after close

* fix lint error

* fix error message issue

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* keep load_or_save_cache when destoryComputation (#698)

* add potential exception handle & open session_option to optimize

* update odla_computation::init() to return odla_status value

* fix popart_config load error

* improve load config logic

* Check the sdk version of the cache

* Format the codes to pass lint checking

* reset other necessary popart config value

* add return value in computation::init()

* clear cache_fs before write new data

* clear fstream after close

* fix lint error

* fix error message issue

* keep load_or_save_cache value

* use popart::core::packageHash for version info

update some log type to err & get sdk packagehash in use_default()

use assign instead of =& remove useless logic

close log

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* update popart api to 2.3.1 & optimize logs (#715)

* update to poplar 2.3.1 & remove useless code

* update log message & add open log option

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>

* cherry-pick qtc optimization updates by jackz (#723)

* update to poplar 2.3.1 & remove useless code

* update log message & add open log option

* Change wait to use condition variable, and change the contextqueues to use mutex

make the format of codes to meet requirements

Correct the == to = for set the buffer_ element to nullptr

Co-authored-by: yanwei <yw01041751@alibaba-inc.com>
Co-authored-by: gcuser <jackz@graphcore.ai>

* add vodla to ipu (#768)

* Add vODLA code from to HALO (#748)

Add vODLA code to HALO

(cherry picked from commit e3e471e)

* Add inference time for both release and debug version as requested from zhiwei.

(cherry picked from commit 6d56f8a)

* [ODLA] Add device arg for LoadExecutable

(cherry picked from commit 0099672)

Co-authored-by: Weiming Zhao <weiming.zhao@alibaba-inc.com>

* Fixed a bug in popart_config.cc std move. (#771)

Fixed a bug in popart_config.cc std move.

* Add the test for the cache configuration (#730)

* Add information to check the cache sdk version and the working sdk version

* Set the log level to info for test

* Add environment variable for error injection test

* format the code for lint check

* Add information to show the ENGION OPTIONS set

* Make it recompile when cache version mismatch

* Remote the environment for error simulator to make it set by environment variable

* Add error injector for temporarily test

* Fix the problem when there's not the error injection config file

* Move the error injector to the odla_CreateComputation to make sure it will be called for each inference request

* Add try catch to computation create and init

* make the error handling code at one place

* Use config file setting for batch per step when it's inconsistent with ODLA set

* set the log level when error injector used

* pass the error code to caller

* Make sdk version mismatch cache recompile when error injector set

* Make the popart log level take POPART_LOG_LEVEL firstly

* change format to pass lint

Co-authored-by: tianboh <54729592+tianboh@users.noreply.github.com>

* [ODLA] Add async call back support

* [ODLA/popart] Add dummy odla_SetContextItem

* Add PIPELINE_ASYNC to support VODLA

* [ODLA][Async] Add status as callback argument

* callback function definition changes, and handle the error raised by poplar SDK

* Move computation init to odla_BindToArgument to avoid problem caused by ASYNC call (#840)

* callback function definition changes, and handle the error raised by poplar SDK

* Move computation init to odla_BindToArgument to avoid early call odla_CreteContext before compuation graph constructed in ASYN call

* Add the mode string for pipeline_async

* Add more debug information for async (#845)

* callback function definition changes, and handle the error raised by poplar SDK

* Move computation init to odla_BindToArgument to avoid early call odla_CreteContext before compuation graph constructed in ASYN call

* Add the mode string for pipeline_async

* Add more debug information for async context

* return the error by callback

* Use pipeline_async to create context when execution mode is UNKNOWN

* Check the execution mode when do the async computation (#866)

* Check the execution mode when do the async computation

* debug core dump problem with vodla device reuse (#868)

* Make all resources used in thread intialized before start the thread

* Add e.what() in print when catch exception. (#877)

* reset the QManager status to ODLA_SUCCESS when create the new queue (#893)

* Remove the duplicate vodla

* change the file to pass lint format check

Co-authored-by: gcuser <gcuser@alibaba-inc.com>
Co-authored-by: youben.yyb <youben.yyb@alibaba-inc.com>
Co-authored-by: yanwei <yanw@graphcore.ai>
Co-authored-by: yw01041751 <yw01041751@alibaba-inc.com>
Co-authored-by: Weiming Zhao <weiming.zhao@alibaba-inc.com>
Co-authored-by: wangchang-2020 <73893382+wangchang-2020@users.noreply.github.com>
Co-authored-by: Zars19 <1036473307@qq.com>
Co-authored-by: tianboh <54729592+tianboh@users.noreply.github.com>
Co-authored-by: yanwei-gr <64010848+yanwei-gr@users.noreply.github.com>

Author: 9 people

ODLA/include/ODLA/odla_compute.h

@@ -61,6 +61,8 @@ typedef enum {
   ODLA_AGGREGATE_OPS,
   ODLA_ENABLE_ENGINE_CACHE,
   ODLA_CACHE_DIR,
+  ODLA_ASYNC_CALLBACK_FUNC,
+  ODLA_ASYNC_CALLBACK_ARG,
 } odla_item_type;
 
 //! \brief BF16 mode

ODLA/platforms/odla_popart/custom_ops/erf.cc

@@ -133,8 +133,7 @@ PostProcessOpx::PostProcessOpx(popart::Op* op, popart::popx::Devicex* devicex)
   numTiles_vtx = 90;
   numTiles_partial = target.getTilesPerIPU() - numTiles_vtx;
   numWorkers_ = target.getNumWorkerContexts();
-  graph().addCodelets(
-      "../../../../ODLA/platforms/odla_popart/custom_ops/codelets_nms.cpp");
+  graph().addCodelets("codelets_nms.cpp");
 }
 
 void PostProcessOpx::grow(poplar::program::Sequence& prog) const {

ODLA/platforms/odla_popart/custom_ops/postprocess.cc

@@ -70,8 +70,7 @@ RsqrtOpx::RsqrtOpx(popart::Op* op, popart::popx::Devicex* devicex)
 void RsqrtOpx::grow(snap::program::Sequence& prog) const {
   auto result =
       popops::map(graph().getPoplarGraph(), popops::expr::UnaryOpType::RSQRT,
-                  getInTensor(0).getPoplarTensor(), prog.getPoplarSequence(),
-                  debugContext());
+                  getInTensor(0).getPoplarTensor(), prog.getPoplarSequence(), debugContext());
   setOutTensor(0, snap::Tensor{result, graph()});
 }