{"id":328,"date":"2019-08-15T08:29:12","date_gmt":"2019-08-15T12:29:12","guid":{"rendered":"http:\/\/bhoey.com\/blog\/?p=328"},"modified":"2020-06-23T12:24:12","modified_gmt":"2020-06-23T16:24:12","slug":"high-availability-rabbitmq-with-mirrored-queues","status":"publish","type":"post","link":"https:\/\/bhoey.com\/blog\/high-availability-rabbitmq-with-mirrored-queues\/","title":{"rendered":"High Availability RabbitMQ With Mirrored Queues"},"content":{"rendered":"

Background<\/h3>\n

RabbitMQ is a robust message queue which features high message throughput, configurable acknowledgements, an intuitive management GUI and wide client library support. Written in Erlang, RabbitMQ has built a reputation for outstanding stability which makes it a popular choice as a core infrastructure system.<\/p>\n

As you plan your overall messaging architecture a universal requirement is to minimize downtime from any single point of failure. Fortunately RabbitMQ comes equipped with built-in high-availability facilities, your tolerance for message loss will determine the which of the available HA options and approaches fits best. This post will mainly focus on setting up RabbitMQ mirrored queues which provide the highest protection against message loss.<\/p>\n

Clustering vs Mirrored Queues<\/h3>\n

For the purpose of this post, its sufficient to know that RabbitMQ clusters do in fact replicate items like users, exchanges, routing keys and queues among member nodes, but importantly they do not replicate the messages themselves1<\/a><\/sup>. In the event of a RabbitMQ node failure, clients can reconnect to a different node, declare their queues and begin to receive published messages from that time on, however any messages that were still waiting in a queue located on the failed node will be lost if the host becomes unrecoverable.<\/p>\n

Mirrored queues introduce an additional level of protection by replicating published messages to other nodes in the cluster as well. In the event of a node failure, another cluster node will assume the queue master role for mirrored queues that were present on the failed node and then proceed to handle messaging operations for these mirrored queues thereafter, including serving messages that were still queued up at the time of the failure. This additional protection however does come at a cost of reduced message throughput and increased server load.<\/p>\n

Requirements<\/h3>\n

Before we can configure mirrored queues we need to first have a RabbitMQ cluster in place.<\/p>\n

\n

RabbitMQ Cluster Requirements<\/h4>\n
    \n
  1. Two or more RabbitMQ nodes<\/li>\n
  2. Identical Erlang cookie value in each node<\/li>\n
  3. Peer discovery method<\/li>\n<\/ol>\n<\/div>\n

    The identical Erlang cookie value<\/b> is necessary for the nodes to authenticate between themselves to form a cluster. Since we will be using docker-compose for this demo we can specify this value as an environment variable and the default RabbitMQ docker-entrypoint.sh script will write this value to the correct file. If you are deploying RabbitMQ to hosts directly or using some other configuration orchestration see the official clustering guide for the location of the cookie file2<\/a><\/sup>.<\/p>\n

    The peer discovery method<\/b> we will use is a hard-coded list of nodes that make up the cluster. This is the simplest method which avoids introducing additional dependencies, making it perfect for short demonstration purposes, but you'll likely want to select a more dynamic peer discovery method outlined in the cluster formation documentation3<\/a><\/sup>.<\/p>\n

    \n

    Mirrored Queues Requirements<\/h4>\n
      \n
    1. RabbitMQ Cluster<\/li>\n
    2. High-availability policy<\/li>\n
    3. One or more queues to apply the HA policy to<\/li>\n<\/ol>\n<\/div>\n

      As mentioned above, mirrored queue functionality is layered on top of the cluster<\/b> mechanisms, so you'll need to set up a cluster if you want to enable mirrored queues.<\/p>\n

      The high-availability policy<\/b> defines the characteristics of the message replication, such as replication batch size, along with a pattern to match against queues you want mirrored. In the demo below we will indicate that the policy should apply to any queues with names starting with 'ha.'.<\/p>\n

      Demo<\/h3>\n

      Demo Overview<\/h4>\n

      For this demo we will spin up two RabbitMQ nodes from scratch in a cluster configuration using docker-compose. We will also leverage the management plugin to pre-define an HA policy, a single queue to be mirrored and other initial configurations so that our cluster has everything it needs to be fully functional from the start.<\/p>\n

      An important thing to note is that the data volume mapping has been commented out in the docker-compose.yml file. This was done since this demo is meant to highlight how to start fresh without any previous state information. When the cluster is running, state information will be persisted to the data volume and will be read in on subsequent starts. Since this demo does not map the data volume, any state information will be lost when the node is shut down<\/b>. In real world use you'll want to map the data volume to an appropriate location on your system to avoid losing data on restart.<\/p>\n

      Demo Files<\/h4>\n

      Below are the files necessary to start up two RabbitMQ nodes using docker-compose. They are intentionally minimalist to highlight the required elements and provide a stable base to allow for experimentation.<\/p>\n

      rabbitmq.conf<\/h5>\n

      The new style rabbitmq.conf file below simply defines the peer discovery method, the nodes participating in the cluster and indicates the definitions file to load on start (this last functionality is provided by the management plugin).<\/p>\n

      \n
      \n\n\n\n
      \n
      \n
      # Load definitions file that includes HA policy<\/code><\/div>\n
      management.load_definitions =<\/code> \/etc\/rabbitmq\/definitions.json<\/code><\/div>\n
      <\/div>\n
      # Declare cluster via host list<\/code><\/div>\n
      cluster_formation.peer_discovery_backend =<\/code> rabbit_peer_discovery_classic_config<\/code><\/div>\n
      cluster_formation.classic_config.nodes.1 =<\/code> rabbit@rabbit1<\/code><\/div>\n
      cluster_formation.classic_config.nodes.2 =<\/code> rabbit@rabbit2<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
      rabbit1-definitions.json<\/h5>\n

      The primary reason we need to use a definition file is to pre-define the HA policy that will apply to queues created in the cluster. An interesting surprise of using a definitions file however is that by using one on a fresh RabbitMQ instance, it will skip the automatic creation of the default 'guest' user. It is therefore necessary to additionally define one or more users explicitly in the definitions file.<\/p>\n

      \n
      \n\n\n\n
      \n
      \n
      {<\/code><\/div>\n
        <\/code>\"vhosts\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"\/\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"users\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"admin\"<\/code>,<\/code><\/div>\n
            <\/code>\"password\":<\/code> \"admin\"<\/code>,<\/code><\/div>\n
            <\/code>\"tags\":<\/code> \"administrator\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"permissions\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"user\":<\/code>\"admin\"<\/code>,<\/code><\/div>\n
            <\/code>\"vhost\":<\/code>\"\/\"<\/code>,<\/code><\/div>\n
            <\/code>\"configure\":<\/code>\".*\"<\/code>,<\/code><\/div>\n
            <\/code>\"write\":<\/code>\".*\"<\/code>,<\/code><\/div>\n
            <\/code>\"read\":<\/code>\".*\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"policies\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"ha-all\"<\/code>,<\/code><\/div>\n
            <\/code>\"apply-to\":<\/code> \"all\"<\/code>,<\/code><\/div>\n
            <\/code>\"definition\":<\/code> {<\/code><\/div>\n
              <\/code>\"ha-mode\":<\/code> \"all\"<\/code>,<\/code><\/div>\n
              <\/code>\"ha-sync-mode\":<\/code> \"automatic\"<\/code>,<\/code><\/div>\n
              <\/code>\"ha-sync-batch-size\":<\/code> 1<\/code><\/div>\n
            <\/code>},<\/code><\/div>\n
            <\/code>\"pattern\":<\/code> \"^ha\\.\"<\/code>,<\/code><\/div>\n
            <\/code>\"priority\":<\/code> 0,<\/code><\/div>\n
            <\/code>\"vhost\":<\/code> \"\/\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"queues\":<\/code><\/div>\n
        <\/code>[<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"ha.queue1\"<\/code>,<\/code><\/div>\n
            <\/code>\"vhost\":<\/code> \"\/\"<\/code>,<\/code><\/div>\n
            <\/code>\"durable\":<\/code> true,<\/code><\/div>\n
            <\/code>\"auto_delete\":<\/code> false,<\/code><\/div>\n
            <\/code>\"arguments\":<\/code> {}<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>]<\/code><\/div>\n
      }<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
      rabbit2-definitions.json<\/h5>\n

      The rabbit2 definitions file can in fact just be an empty file as it will sync the configurations from the rabbit1 node when joining the cluster. However to err on the side of caution I prefer to include a user and permissions as well as the HA policy definition in this file in case the cluster gets in some kind of split-brain state that would need manual intervention.<\/p>\n

      It should be noted what is not<\/u><\/b> defined in the rabbit2 file is the 'ha.queue1' definition since whichever node a queue is declared on becomes that queue's master node.<\/p>\n

      \n
      \n\n\n\n
      \n
      \n
      {<\/code><\/div>\n
        <\/code>\"vhosts\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"\/\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"users\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"admin\"<\/code>,<\/code><\/div>\n
            <\/code>\"password\":<\/code> \"admin\"<\/code>,<\/code><\/div>\n
            <\/code>\"tags\":<\/code> \"administrator\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"permissions\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"user\":<\/code>\"admin\"<\/code>,<\/code><\/div>\n
            <\/code>\"vhost\":<\/code>\"\/\"<\/code>,<\/code><\/div>\n
            <\/code>\"configure\":<\/code>\".*\"<\/code>,<\/code><\/div>\n
            <\/code>\"write\":<\/code>\".*\"<\/code>,<\/code><\/div>\n
            <\/code>\"read\":<\/code>\".*\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>],<\/code><\/div>\n
      <\/div>\n
        <\/code>\"policies\":<\/code> [<\/code><\/div>\n
          <\/code>{<\/code><\/div>\n
            <\/code>\"name\":<\/code> \"ha-all\"<\/code>,<\/code><\/div>\n
            <\/code>\"apply-to\":<\/code> \"all\"<\/code>,<\/code><\/div>\n
            <\/code>\"definition\":<\/code> {<\/code><\/div>\n
              <\/code>\"ha-mode\":<\/code> \"all\"<\/code>,<\/code><\/div>\n
              <\/code>\"ha-sync-mode\":<\/code> \"automatic\"<\/code>,<\/code><\/div>\n
              <\/code>\"ha-sync-batch-size\":<\/code> 1<\/code><\/div>\n
            <\/code>},<\/code><\/div>\n
            <\/code>\"pattern\":<\/code> \"^ha\\.\"<\/code>,<\/code><\/div>\n
            <\/code>\"priority\":<\/code> 0,<\/code><\/div>\n
            <\/code>\"vhost\":<\/code> \"\/\"<\/code><\/div>\n
          <\/code>}<\/code><\/div>\n
        <\/code>]<\/code><\/div>\n
      }<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
      docker-compose.yml<\/h5>\n

      The docker-compose.yml file brings everything together under one umbrella. It defines both RabbitMQ instances, the files and ports to map, and the shared Erlang cookie value.<\/p>\n

      One thing to note is that I've customized rabbit2's 'command' value in order to stagger the startup of the two nodes to avoid a race condition where neither node believes its the primary4<\/a><\/sup>.<\/p>\n

      \n
      \n\n\n\n
      \n
      \n
      version:<\/code> \"2\"<\/code><\/div>\n
      <\/div>\n
      services:<\/code><\/div>\n
          <\/code>rabbit1:<\/code><\/div>\n
              <\/code>image:<\/code> rabbitmq<\/code>:<\/code>3-management<\/code><\/div>\n
              <\/code>container_name:<\/code> rabbit1<\/code><\/div>\n
              <\/code>hostname:<\/code> rabbit1<\/code><\/div>\n
              <\/code>ports:<\/code><\/div>\n
                  <\/code>-<\/code> 5672<\/code>:<\/code>5672<\/code><\/div>\n
                  <\/code>-<\/code> 15672<\/code>:<\/code>15672<\/code><\/div>\n
              <\/code>environment:<\/code><\/div>\n
                  <\/code>-<\/code> RABBITMQ_ERLANG_COOKIE=shared_cookie<\/code><\/div>\n
              <\/code>volumes:<\/code><\/div>\n
                  <\/code>-<\/code> $PWD\/rabbit1-definitions.json<\/code>:<\/code>\/etc\/rabbitmq\/definitions.json<\/code>:<\/code>ro<\/code><\/div>\n
                  <\/code>-<\/code> $PWD\/rabbitmq.conf<\/code>:<\/code>\/etc\/rabbitmq\/rabbitmq.conf<\/code>:<\/code>ro<\/code><\/div>\n
                <\/code># - $PWD\/rabbit1-data:\/var\/lib\/rabbitmq<\/code><\/div>\n
      <\/div>\n
          <\/code>rabbit2:<\/code><\/div>\n
              <\/code>image:<\/code> rabbitmq<\/code>:<\/code>3-management<\/code><\/div>\n
              <\/code>container_name:<\/code> rabbit2<\/code><\/div>\n
              <\/code>hostname:<\/code> rabbit2<\/code><\/div>\n
              <\/code>ports:<\/code><\/div>\n
                  <\/code>-<\/code> 5673<\/code>:<\/code>5672<\/code><\/div>\n
                  <\/code>-<\/code> 15673<\/code>:<\/code>15672<\/code><\/div>\n
              <\/code>environment:<\/code><\/div>\n
                  <\/code>-<\/code> RABBITMQ_ERLANG_COOKIE=shared_cookie<\/code><\/div>\n
              <\/code>volumes:<\/code><\/div>\n
                  <\/code>-<\/code> $PWD\/rabbit2-definitions.json<\/code>:<\/code>\/etc\/rabbitmq\/definitions.json<\/code>:<\/code>ro<\/code><\/div>\n
                  <\/code>-<\/code> $PWD\/rabbitmq.conf<\/code>:<\/code>\/etc\/rabbitmq\/rabbitmq.conf<\/code>:<\/code>ro<\/code><\/div>\n
                <\/code># - $PWD\/rabbit2-data:\/var\/lib\/rabbitmq<\/code><\/div>\n
              <\/code>command:<\/code> sh -c <\/code>'sleep 10 && rabbitmq-server'<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

      Running The Demo<\/h4>\n

      Once the files are in place (and assuming you have docker-compose installed), its just a matter of running 'docker-compose up':<\/p>\n

      \n
      \n\n\n\n
      \n
      \n
      $ docker-compose up<\/code><\/div>\n
      <\/div>\n
      Starting <\/code>rabbit1<\/code> ... done<\/code><\/div>\n
      Starting <\/code>rabbit2<\/code> ... done<\/code><\/div>\n
      Attaching to <\/code>rabbit1<\/code>, <\/code>rabbit2<\/code><\/div>\n
      ...<\/code><\/div>\n
      rabbit1    | 2019-08-15 01:59:39.320 [info] <0.271.0> <\/code><\/div>\n
      rabbit1    |  Starting RabbitMQ 3.7.16 on Erlang 22.0.7<\/code><\/div>\n
      ...<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.050 [info] <0.296.0> <\/code><\/div>\n
      rabbit2    |  Starting RabbitMQ 3.7.16 on Erlang 22.0.7<\/code><\/div>\n
      ...<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.476 [info] <0.447.0> Mirrored queue 'ha.queue1' in vhost '\/': Adding mirror on node rabbit@rabbit1: <8837.647.0><\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.485 [info] <0.447.0> Mirrored queue 'ha.queue1' in vhost '\/': Synchronising: 0 messages to synchronise<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.485 [info] <0.447.0> Mirrored queue 'ha.queue1' in vhost '\/': Synchronising: batch size: 4096<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.486 [info] <0.296.0> Running boot step load_definitions defined by app rabbitmq_management<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.486 [info] <0.466.0> Mirrored queue 'ha.queue1' in vhost '\/': Synchronising: all slaves already synced<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.486 [info] <0.296.0> Applying definitions from \/etc\/rabbitmq\/definitions.json<\/code><\/div>\n
      ...<\/code><\/div>\n
      rabbit1    | 2019-08-15 01:59:49.500 [info] <0.385.0> rabbit on node rabbit@rabbit2 up<\/code><\/div>\n
      rabbit2    | 2019-08-15 01:59:49.500 [info] <0.398.0> rabbit on node rabbit@rabbit1 up<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

      Verify Setup<\/h4>\n

      After browsing to the admin GUI at http:\/\/localhost:15672<\/a> and logging in with admin\/admin you should see the following:<\/p>\n\n\n\n
      \"nodes<\/a><\/td>\n\"the<\/a><\/td>\n\"mirrored<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Client code<\/h4>\n

      The client code setup is slightly different for connecting to mirrored queues, you'll want to indicate to the RabbitMQ\/AMQP library that there are multiple cluster nodes so that in the event of a failure they can attempt to connect to another node.<\/p>\n

      The shuffling of the node list in both code snippets is to distribute clients across nodes. Keep in mind that the node where the queue was defined (ie the queue master) will still handle all messages routing for that queue.<\/p>\n

      Python<\/h5>\n
      #!\/usr\/bin\/env python3\nimport pika\nimport random\n\ncreds = pika.PlainCredentials('admin', 'admin')\nparams = [\n           pika.ConnectionParameters(host='localhost', port=5672, credentials=creds),\n           pika.ConnectionParameters(host='localhost', port=5673, credentials=creds),\n         ]\n\nrandom.shuffle(params)\n\nconnection = pika.BlockingConnection(params)\nchannel = connection.channel()\n\n# Publish message\nchannel.basic_publish(\n                       exchange='',\n                       routing_key='ha.queue1',\n                       body='Test message'\n                     )\n\n# Receive message\nresp_get_ok, resp_props, resp_body = channel.basic_get(queue='ha.queue1', auto_ack=True)\nprint(resp_body)\n\nchannel.close()\nconnection.close()\n<\/pre>\n
      Java<\/h5>\n
      import com.rabbitmq.client.ConnectionFactory;\nimport com.rabbitmq.client.Connection;\nimport com.rabbitmq.client.Channel;\nimport com.rabbitmq.client.Address;\nimport com.rabbitmq.client.AMQP.BasicProperties;\nimport com.rabbitmq.client.GetResponse;\nimport java.util.Arrays;\nimport java.util.List;\nimport java.util.Collections;\n\npublic class RabbitMQTest{\n\n    public static void main(String[] argv) throws Exception {\n\n        List<Address> nodes = Arrays.asList(\n                                 new Address(\"localhost\", 5672),\n                                 new Address(\"localhost\", 5673)\n                              );\n\n        Collections.shuffle(nodes);\n\n        ConnectionFactory factory = new ConnectionFactory();\n        factory.setUsername(\"admin\");\n        factory.setPassword(\"admin\");\n\n        Connection connection = factory.newConnection(nodes);\n        Channel channel = connection.createChannel();\n\n        \/\/ Publish message\n        String exch = \"\";\n        BasicProperties props = null;\n        String routing_key = \"ha.queue1\";\n        String msg = \"Test message\";\n        channel.basicPublish(exch, routing_key, props, msg.getBytes());\n\n        \/\/ Receive message\n        String queue = \"ha.queue1\";\n        boolean autoAck = true;\n        GetResponse response = channel.basicGet(queue, autoAck);\n        byte[] body = response.getBody();\n        System.out.println(new String(body));\n\n        channel.close();\n        connection.close();\n    }\n}\n<\/pre>\n
      References<\/h3>\n
        \n
      1. <\/a>What is Replicated?: https:\/\/www.rabbitmq.com\/clustering.html#overview-what-is-replicated<\/a><\/li>\n
      2. <\/a>Clustering Guide: https:\/\/www.rabbitmq.com\/clustering.html<\/a><\/li>\n
      3. <\/a>Cluster Formation and Peer Discovery: https:\/\/www.rabbitmq.com\/cluster-formation.html<\/a><\/li>\n
      4. <\/a>Race Conditions During Initial Cluster Formation: https:\/\/www.rabbitmq.com\/cluster-formation.html#initial-formation-race-condition<\/a><\/li>\n
      5. Highly Available (Mirrored) Queues: https:\/\/www.rabbitmq.com\/ha.html<\/a><\/li>\n
      6. Loading Definitions (Schema) at Startuphttps:\/\/www.rabbitmq.com\/management.html#load-definitions<\/a><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
        RabbitMQ is a robust message queue which features high message throughput, configurable acknowledgements, an intuitive management GUI and wide client library support. Written in Erlang, RabbitMQ has built a reputation for outstanding stability which makes it a popular choice as a core infrastructure system.<\/p>\n
        As you plan your overall messaging architecture a universal requirement is to minimize downtime from any single point of failure. Fortunately RabbitMQ comes equipped with built-in high-availability facilities, your tolerance for message loss will determine the which of the available HA options and approaches fits best. This post will mainly focus on setting up RabbitMQ mirrored queues which provide the highest protection against message loss. [Continue reading...] High Availability RabbitMQ With Mirrored Queues<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":804,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[54],"tags":[15,57,55,56,53],"_links":{"self":[{"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/posts\/328"}],"collection":[{"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/comments?post=328"}],"version-history":[{"count":143,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/posts\/328\/revisions"}],"predecessor-version":[{"id":809,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/posts\/328\/revisions\/809"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/media\/804"}],"wp:attachment":[{"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/media?parent=328"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/categories?post=328"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bhoey.com\/blog\/wp-json\/wp\/v2\/tags?post=328"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}