腾讯云计算加速套件TACO KitTensorFlow 模型推理部署

操作场景

本文介绍如何使用 TACO Infer 部署模型。在部署前，请确保您已完成 TensorFlow 模型优化，并且验证模型的性能和正确性符合预期后，您即可通过本文将模型部署在实际生产环境中。

操作步骤

环境准备

服务器：参见 TACO Infer 安装，选购 CPU 机型。ABI 版本：TACO Infer 支持 CXX11 ABI。如有其他版本需求请通过 联系我们 获取支持。SDK 包安装：在开发部署模型之前，请确保您已经安装了 TACO Infer SDK 安装包，详情请参见 获取 Wheel 包及 SDK 包。解压后可查看安装包中包含三个动态链接库和一个可执行文件：

[root@VM-3-46-centos inceptionv3]# ll lib/total 416180-rwxr-xr-x 1 root root   1018440 Mar 31 20:25 libomp-1fdec59b.so-rwxr-xr-x 1 root root  42617800 Mar 31 20:25 libtaco_tf.so-rwxr-xr-x 1 root root 125954112 Mar 31 20:25 libtidy_ops.so-rwxr-xr-x 1 root root 256572616 Mar 31 20:25 tidy_vm

说明建议您将所有 TACO 库文件拷贝到系统库目录 /usr/lib 下，以便链接器 ld 进行链接时可定位。或您也可以将 TACO 库文件拷贝到其他路径，并在 LD_LIBRARY_PATH 环境变量中添加库所在路径。请确保所有的 TACO 库文件位于同一路径下。

推理代码开发

以下代码以 TensorFlow C++ API 展示优化后模型的加载运行过程，您只需要按照标准的 TensorFlow C++ API 加载经过优化的模型即可，和加载普通的 TF 模型没有任何区别。

#include #include 
#include "tensorflow/core/framework/graph.pb.h"#include "tensorflow/core/framework/tensor.h"#include "tensorflow/core/graph/default_device.h"#include "tensorflow/core/graph/graph_def_builder.h"#include "tensorflow/core/lib/core/threadpool.h"#include "tensorflow/core/lib/strings/stringprintf.h"#include "tensorflow/core/platform/init_main.h"#include "tensorflow/core/platform/logging.h"#include "tensorflow/core/platform/types.h"#include "tensorflow/core/public/session.h"#include "tensorflow/core/public/session_options.h"#include "tensorflow/cc/client/client_session.h"#include "tensorflow/core/protobuf/meta_graph.pb.h"#include "tensorflow/c/c_api.h"
using tensorflow::GraphDef;using tensorflow::int32;using tensorflow::string;
void LoadFrozenPbModel(    const std::unique_ptr& session,    std::string model_path,    bool as_text) {  GraphDef graph_def;  tensorflow::Status load_graph_status;  std::cout << "Model Path: " << model_path << std::endl;  if (as_text) {    load_graph_status =        ReadTextProto(tensorflow::Env::Default(), model_path, &graph_def);  } else {    load_graph_status =        ReadBinaryProto(tensorflow::Env::Default(), model_path, &graph_def);  }
  if (!load_graph_status.ok()) {      std::cout << "Failed to load model: " << model_path                    << load_graph_status.ToString() << std::endl;  }
  auto session_status = session->Create(graph_def);  if (!session_status.ok()) {      std::cout << "Failed to create session" << session_status.ToString() << std::endl;  }}
void RunInference(    const std::string& model_path,    const std::vector<std::pair>& inputs,    const std::vector& output_tensor_names,    const std::vector& target_node_names,    std::vector& outputs) {  // NOLINT  // Create session  tensorflow::SessionOptions options;  std::unique_ptr session(tensorflow::NewSession(options));  // Load model  LoadFrozenPbModel(session, model_path, false);
  TF_CHECK_OK(      session->Run(inputs, output_tensor_names, target_node_names, &outputs));}
int main() {  std::string model_path = "./optimized_model/fast-transformer-encoder.pb";
  // Create test input data  std::vector<std::pair> inputs;  auto input_tensor = tensorflow::Tensor(      tensorflow::DT_FLOAT, tensorflow::TensorShape({1, 32, 768}));  auto flat = input_tensor.flat();  for (int i = 0; i < 24576; i++) {    flat(i) = 0.5;  }  string input_name = "Placeholder";  inputs.emplace_back(input_name, input_tensor);
  // Create output tensor  const std::vector output_tensor_names = {"mul_1:0"};  const std::vector target_node_names = {};  std::vector outputs;
  RunInference(      model_path, inputs, output_tensor_names, target_node_names, outputs);
  std::cout << "Output tensor: [" << std::endl;  for (int i = 0; i < 10; i++) {      std::cout << outputs[0].flat()(i) << std::endl;  }  std::cout << "]" << std::endl;}

编译链接

编译以上代码时，需要链接 Taco 提供的 libtaco_tf.so 和 libtidy_ops.so 两个动态库，及链接 TensorFlow 提供的 libtensorflow_framework.so 和 libtensorflow_cc.so 两个动态链接库。其中，libtensorflow_framework.so 位于 TensorFlow 的 Python 安装目录中。而 libtensorflow_cc.so 没有随着TensorFlow Python 安装包一起发行，需要您下载 TensorFlow 源码自行编译。1. 执行以下命令，完成编译。

git clone https://github.com/tensorflow/tensorflowcd tensorflowgit checkout ${xxx version}./configurebazel build --config=opt //tensorflow:libtensorflow_cc.so

2. libtensorflow_cc.so 编译完成后，即可进行模型部署代码的编译。编译参数如下所示：

#!/bin/bash
gcc -std=c++11 \ -I/root/taco_test/1.15.0/include \ -L/root/taco_test/venv/taco_dev/lib64/python3.6/site-packages/tensorflow_core -ltensorflow_framework \ -L/root/taco_test/lib -ltensorflow_cc -ltaco_tf -ltidy_ops \ -lstdc++ \ -o tf_sdk_demo tf_sdk_demo.cc

编译完成后，得到一个可运行的二进制文件。如下所示：

[root@VM-3-56-ubuntu (Taco Dev) /home/ubuntu/taco_test/test_taco_sdk_demo]#ll-rw-rw-r--  1  500  500 3.4K Mar 29 16:04 taco_sdk_demo.cc-rwxr-xr-x  1 root root  99K Mar 29 16:04 taco_sdk_demo*drwxr-xr-x  2 root root 4.0K Mar 29 16:06 ./drwxrwxr-x 10  500  500 4.0K Mar 29 16:58 ../

3. 确认相关的动态链接库已经正确链接：

(taco_dev) [root@VM-3-46-centos fast_transformer_encoder]# ldd tf_sdk_demo linux-vdso.so.1 =>  (0x00007fff34bea000) libtensorflow_framework.so.1 => /root/taco_test/venv/taco_dev/lib64/python3.6/site-packages/tensorflow_core/libtensorflow_framework.so.1 (0x00007efff59ab000) libtensorflow_cc.so.1 => /usr/local/lib/libtensorflow_cc.so.1 (0x00007effee66f000) libtaco_tf.so => /usr/local/lib/libtaco_tf.so (0x00007effec488000) libtidy_ops.so => /usr/local/lib/libtidy_ops.so (0x00007effea335000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007effea02d000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007effe9e17000) libc.so.6 => /lib64/libc.so.6 (0x00007effe9a49000) librt.so.1 => /lib64/librt.so.1 (0x00007effe9841000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007effe9625000) libdl.so.2 => /lib64/libdl.so.2 (0x00007effe9421000) libm.so.6 => /lib64/libm.so.6 (0x00007effe911f000) /lib64/ld-linux-x86-64.so.2 (0x00007efff76b7000) libomp-1fdec59b.so => /usr/local/lib/libomp-1fdec59b.so (0x00007effe8e4e000) libcurl.so.4 => /lib64/libcurl.so.4 (0x00007effe8be4000) libuuid.so.1 => /lib64/libuuid.so.1 (0x00007effe89df000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007effe857c000) libidn.so.11 => /lib64/libidn.so.11 (0x00007effe8349000) libssh2.so.1 => /lib64/libssh2.so.1 (0x00007effe811c000) libssl3.so => /lib64/libssl3.so (0x00007effe7eb9000) libsmime3.so => /lib64/libsmime3.so (0x00007effe7c91000) libnss3.so => /lib64/libnss3.so (0x00007effe7958000) libnssutil3.so => /lib64/libnssutil3.so (0x00007effe7728000) libplds4.so => /lib64/libplds4.so (0x00007effe7524000) libplc4.so => /lib64/libplc4.so (0x00007effe731f000) libnspr4.so => /lib64/libnspr4.so (0x00007effe70e1000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007effe6e94000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007effe6bab000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007effe6978000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007effe6774000) liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007effe6565000) libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007effe6310000) libz.so.1 => /lib64/libz.so.1 (0x00007effe60fa000) libssl.so.10 => /lib64/libssl.so.10 (0x00007effe5e88000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007effe5c78000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007effe5a74000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007effe585a000) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007effe563d000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007effe5416000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007effe51df000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007effe4f7d000) libfreebl3.so => /lib64/libfreebl3.so (0x00007effe4d7a000)

推理计算

部署程序编译完成后，运行即可加载优化后的模型并进行推理计算。可查看模型正常加载运行，并输出了推理计算结果。如下所示：

[root@VM-3-56-ubuntu (Taco Dev) /home/ubuntu/taco_test/test_taco_sdk_demo]#./taco_sdk_demo
2022-03-31 20:52:21.702558: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2022-03-31 20:52:21.703465: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (VM-3-46-centos): /proc/driver/nvidia/version does not existModel Path: ./optimized_model/fast-transformer-encoder.pb...Output tensor: [0.692124-1.309811.93331-0.0825812-0.423409-0.73291-2.133660.758448-1.251490.659645]



官网1折活动，限时活动，即将结束，速速收藏
同尘科技为腾讯云授权服务中心。
购买腾讯云产品享受折上折，更有现金返利。同意关联立享优惠