容器化运行机器学习 TensorFlow Serving

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out of the box integration with TensorFlow models, but can be easily extended to serve other types of models.

Dockerized TensorFlow Serving

docker pull tensorflow/serving:latestdocker pull tensorflow/serving:latest-gpu

准备model

export_path_base = sys.argv[-1]export_path = os.path.join(      compat.as_bytes(export_path_base),      compat.as_bytes(str(FLAGS.model_version)))print 'Exporting trained model to', export_pathbuilder = tf.saved_model.builder.SavedModelBuilder(export_path)builder.add_meta_graph_and_variables(      sess, [tag_constants.SERVING],      signature_def_map={           'predict_images':               prediction_signature,           signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:               classification_signature,      },      legacy_init_op=legacy_init_op)builder.save()

assets/assets.extra/variables/    variables.data-?????-of-?????    variables.indexsaved_model.pb

运行服务

sudo -H pip install docker-compose

version: "3"
services:  serving:    image: tensorflow/serving:latest    restart: unless-stopped


    
    ports:      - 8500:8500      - 8501:8501    volumes:      - ./model0:/models/MODEL0    environment:      - MODEL_NAME=MODEL0      - NVIDIA_VISIBLE_DEVICES=0

docker-compose up -d

GPU预留方案

deploy:      replicas: 2      resources:        reservations:          generic_resources:          - discrete_resource_spec:              kind: 'gpu'              value: 1

serve成功的log示例

$ docker-compose logs -fserving_1  | 2018-11-12 11:45:29.679213: I tensorflow_serving/model_servers/main.cc:157] Building single TensorFlow model file config:  model_name: MODEL0 model_base_path: /models/MODEL0serving_1  | 2018-11-12 11:45:29.679673: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.serving_1  | 2018-11-12 11:45:29.679700: I tensorflow_serving/model_servers/server_core.cc:517]  (Re-)adding model: MODEL0serving_1  | 2018-11-12 11:45:29.780031: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: MODEL0 version: 1}serving_1  | 2018-11-12 11:45:29.780088: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: MODEL0 version: 1}serving_1  | 2018-11-12 11:45:29.780101: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: MODEL0 version: 1}serving_1  | 2018-11-12 11:45:29.780127: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:360] Attempting to load native SavedModelBundle in bundle-shim from: /models/MODEL0/1


    
serving_1  | 2018-11-12 11:45:29.780140: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/MODEL0/1serving_1  | 2018-11-12 11:45:29.847226: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }serving_1  | 2018-11-12 11:45:29.862407: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMAserving_1  | 2018-11-12 11:45:29.920827: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:113] Restoring SavedModel bundle.serving_1  | 2018-11-12 11:45:29.920907: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:123] The specified SavedModel has no variables; no checkpoints were restored.serving_1  | 2018-11-12 11:45:29.920917: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:148] Running LegacyInitOp on SavedModel bundle.serving_1  | 2018-11-12 11:45:29.920932: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:233] SavedModel load for tags { serve }; Status: success. Took 140784 microseconds.serving_1  | 2018-11-12 11:45:29.920969: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:83] No warmup data file found at /models/MODEL0/1/assets.extra/tf_serving_warmup_requestsserving_1  | 2018-11-12 11:45:29.921085: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: MODEL0 version: 1}serving_1  | 2018-11-12 11:45:29.922849: I tensorflow_serving/model_servers/main.cc:327] Running ModelServer at 0.0.0.0:8500 ...serving_1  | [warn] getaddrinfo: address family for nodename not supportedserving_1  | 2018-11-12 11:45:29.924139: I tensorflow_serving/model_servers/main.cc:337] Exporting HTTP/REST API at:localhost:8501 ...serving_1  | [evhttp_server.cc : 235] RAW: Entering the event loop ...

使用

service PredictionService {  // Classify.  rpc Classify(ClassificationRequest) returns (ClassificationResponse);
  // Regress.  rpc Regress(RegressionRequest) returns (RegressionResponse);
  // Predict -- provides access to loaded TensorFlow model.  rpc Predict(PredictRequest) returns (PredictResponse);
  // MultiInference API for multi-headed models.  rpc MultiInference(MultiInferenceRequest) returns (MultiInferenceResponse);
  // GetModelMetadata - provides access to metadata for loaded models.  rpc GetModelMetadata(GetModelMetadataRequest)      returns (GetModelMetadataResponse);}

POST http://host:port/:
URI: /v1/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]


    
VERB: classify|regress|predict

内容为JSON，classify和regress的示例如下（predict略）：

{  // Optional: serving signature to use.  // If unspecifed default serving signature is used.  "signature_name": <string>,
  // Optional: Common context shared by all examples.  // Features that appear here MUST NOT appear in examples (below).  "context": {    "": <value>|    "": <value>|  },
  // List of Example objects  "examples": [    {      // Example 1      "": <value>|,      "": <value>|,      ...    },    {      // Example 2      "": <value>|,      "": <value>|,      ...    }    ...  ]}

Request发送成功后，Response大致如下。

{  "result": [    // List of class label/score pairs for first Example (in request)    [ [, ], [, ], ... ],
    // List of class label/score pairs for next Example (in request)    [ [, ], [, ], ... ],    ...  ]}

{  // One regression value for each example in the request in the same order.  "result": [ , , , ...]}

参考

• tensorflow/serving - Docker Hub
• Using TensorFlow Serving via Docker
• tensorflow/README.md at master · tensorflow/tensorflow
• Serving a TensorFlow Model | TensorFlow
• RESTful API | TensorFlow
• Added support of Generic resources in compose file · docker/cli@1ff73f8
• docker service create | Docker Documentation

• Dockerfile 目前可扩展的语法

• 小白都能看懂的 Kubernetes安全之API-server安全

• 项目中常用的19条MySQL优化技巧

• 被HTTP/2漏洞拖累，所有Kubernetes版本受影响

• Linux基金会主席Thomas：容器该不该放到VM中？

• Serverless 的喧哗与骚动