Streaming ros bags from azure blob storage.
This package provides an implementation of a subset of the rosbag api and the rosbag binary allowing streaming ros bags directly from an azure blob storage without prior downloading the bag.
Summarize contents of a bag
rosbaz info https://contosoaccount.blob.core.windows.net/contosocontainer/my.bagPlay back contents of a bag
rosbaz play --paused https://contosoaccount.blob.core.windows.net/contosocontainer/my.bagNote that only those topics with connected subscribers will actually be downloaded from azure.
We tried to keep the API as close to the rosbag api as possible.
#include "rosbaz/io/az_reader.h"
#include "rosbaz/io/az_writer.h
#include <rosbaz/bag.h>
#include <rosbaz/view.h>
#include <std_msgs/Int32.h>
auto in_url = rosbaz::AzBlobUrl::parse("https://contosoaccount.blob.core.windows.net/contosocontainer/my.bag?SAS_TOKEN");
auto az_reader = std::make_shared<rosbaz::io::AzReader>(in_url);
auto in_bag = rosbaz::Bag::read(az_reader);
for(const auto m : rosbaz::View(in_bag))
{
std_msgs::Int32::ConstPtr i = m.instantiate<std_msgs::Int32>();
if (i != nullptr) {
std::cout << i->data << std::endl;
}
}
auto out_url = rosbaz::AzBlobUrl::parse("https://contosoaccount.blob.core.windows.net/contosocontainer/other.bag?SAS_TOKEN");
auto az_writer = std::make_shared<rosbaz::io::AzWriter>(out_url);
auto out_bag = rosbaz::Bag::write(az_writer);
for(const auto m : rosbaz::View(in_bag))
{
out_bag.write(m.getTopic(), m.getTime(), m);
}
out_bag.close();The following modes are supported
- Azure Cli Credential
az login
rosbaz info https://contosoaccount.blob.core.windows.net/contosocontainer/my.bag- Bearer Token
export AZ_TOKEN=$(az account get-access-token --resource https://storage.azure.com/ -o tsv --query accessToken)
rosbaz info https://contosoaccount.blob.core.windows.net/contosocontainer/my.bag --token $AZ_TOKEN- SAS Token
rosbaz info https://contosoaccount.blob.core.windows.net/contosocontainer/my.bag?SAS_TOKEN- API Key
rosbaz info https://contosoaccount.blob.core.windows.net/contosocontainer/my.bag --account-key $ACCOUNT_KEY- rosbag format 2.0
- no support for encryption
- no support for compressed chunks
- Install prerequsites
sudo apt install -y libssl-dev libcurl4-openssl-dev cmake g++ uuid-dev libxml2-dev- Make sure you have a sufficiently recent cmake, you can install the latest version by
sudo apt install software-properties-common
curl -s https://apt.kitware.com/keys/kitware-archive-latest.asc | sudo apt-key add -
sudo apt-add-repository "deb https://apt.kitware.com/ubuntu/ $(lsb_release -sc) main"
sudo apt install cmake- Install azure-storage-blobs
git clone --branch azure-storage-blobs_12.9.0 https://github.com/Azure/azure-sdk-for-cpp.git
cd azure-sdk-for-cpp
mkdir build.release
cd build.release
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON
cmake --build . --parallel
sudo cmake --install .mkdir rosbaz
cd rosbaz
git clone https://github.com/janbernloehr/rosbaz.git src
catkin build rosbaz