Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FSTORE-1346] Update doc dependencies #197

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions .github/workflows/mkdocs-main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,17 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: set dev version
working-directory: ./java
run: echo "DEV_VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -Ev 'Download|INFO|WARNING')" >> $GITHUB_ENV

- uses: actions/setup-python@v2
- uses: actions/setup-python@v5
with:
python-version: '3.8'
python-version: "3.10"

- name: install deps
working-directory: ./python
Expand All @@ -33,4 +33,3 @@ jobs:

- name: mike deploy docs
run: mike deploy ${{ env.DEV_VERSION }} dev -u

14 changes: 5 additions & 9 deletions .github/workflows/mkdocs-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,14 @@ name: mkdocs-release

on:
push:
branches: [ branch-* ]
branches: [branch-*]

jobs:
publish-release:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
fetch-depth: 0

Expand All @@ -20,16 +20,13 @@ jobs:
- name: set major/minor release version
run: echo "MAJOR_VERSION=$(echo $RELEASE_VERSION | sed 's/^\([0-9]*\.[0-9]*\).*$/\1/')" >> $GITHUB_ENV

- uses: actions/setup-python@v2
- uses: actions/setup-python@v5
with:
python-version: '3.8'
python-version: "3.10"

- name: install deps
working-directory: ./python
run: cp ../README.md . && pip3 install pip==22.0.3 && pip3 install -e .[dev,docs]

- name: use dev mike
run: pip3 uninstall -y mike && pip3 install git+'https://github.com/jimporter/mike.git'
run: cp ../README.md . && pip3 install -e .[dev,docs]

- name: generate autodoc
run: python3 auto_doc.py
Expand All @@ -43,4 +40,3 @@ jobs:
run: |
mike deploy ${{ env.RELEASE_VERSION }} ${{ env.MAJOR_VERSION }} -u --push
mike alias ${{ env.RELEASE_VERSION }} latest -u --push

192 changes: 96 additions & 96 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
## Python development setup

---

- Fork and clone the repository
Expand All @@ -7,58 +8,57 @@

- Install repository in editable mode with development dependencies:

```bash
cd python
pip install -e ".[dev]"
```
```bash
cd python
pip install -e ".[dev]"
```

- Install [pre-commit](https://pre-commit.com/) and then activate its hooks. pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. The library uses pre-commit to ensure code-style and code formatting through [black](https://github.com/psf/black) and [flake8](https://gitlab.com/pycqa/flake8). Run the following commands from the `python` directory:

```bash
cd python
pip install --user pre-commit
pre-commit install
```
```bash
cd python
pip install --user pre-commit
pre-commit install
```

Afterwards, pre-commit will run whenever you commit.

- To run formatting and code-style separately, you can configure your IDE, such as VSCode, to use black and flake8, or run them via the command line:

```bash
cd python
flake8 hopsworks
black hopsworks
```
```bash
cd python
flake8 hopsworks
black hopsworks
```

### Python documentation

We follow a few best practices for writing the Python documentation:

1. Use the google docstring style:

```python
"""[One Line Summary]
```python
"""[One Line Summary]

[Extended Summary]
[Extended Summary]

[!!! example
import xyz
]
[!!! example
import xyz
]

# Arguments
arg1: Type[, optional]. Description[, defaults to `default`]
arg2: Type[, optional]. Description[, defaults to `default`]
# Arguments
arg1: Type[, optional]. Description[, defaults to `default`]
arg2: Type[, optional]. Description[, defaults to `default`]

# Returns
Type. Description.
# Returns
Type. Description.

# Raises
Exception. Description.
"""
```

If Python 3 type annotations are used, they are inserted automatically.
# Raises
Exception. Description.
"""
```

If Python 3 type annotations are used, they are inserted automatically.

2. Hopsworks entity engine methods (e.g. ExecutionEngine etc.) only require a single line docstring.
3. Private REST Api implementations (e.g. GitRemoteApi etc.) should be fully documented with docstrings without defaults.
Expand All @@ -69,42 +69,42 @@ We follow a few best practices for writing the Python documentation:
We use `mkdocs` together with `mike` ([for versioning](https://github.com/jimporter/mike/)) to build the documentation and a plugin called `keras-autodoc` to auto generate Python API documentation from docstrings.

**Background about `mike`:**
`mike` builds the documentation and commits it as a new directory to the gh-pages branch. Each directory corresponds to one version of the documentation. Additionally, `mike` maintains a json in the root of gh-pages with the mappings of versions/aliases for each of the directories available. With aliases you can define extra names like `dev` or `latest`, to indicate stable and unstable releases.
`mike` builds the documentation and commits it as a new directory to the gh-pages branch. Each directory corresponds to one version of the documentation. Additionally, `mike` maintains a json in the root of gh-pages with the mappings of versions/aliases for each of the directories available. With aliases you can define extra names like `dev` or `latest`, to indicate stable and unstable releases.

1. Currently we are using our own version of `keras-autodoc`

```bash
pip install git+https://github.com/moritzmeister/keras-autodoc@split-tags-properties
```
```bash
pip install git+https://github.com/logicalclocks/keras-autodoc
```

2. Install HOPSWORKS with `docs` extras:

```bash
pip install -e .[dev,docs]
```
```bash
pip install -e .[dev,docs]
```

3. To build the docs, first run the auto doc script:

```bash
cd ..
python auto_doc.py
```
```bash
cd ..
python auto_doc.py
```

##### Option 1: Build only current version of docs

4. Either build the docs, or serve them dynamically:

Note: Links and pictures might not resolve properly later on when checking with this build.
The reason for that is that the docs are deployed with versioning on docs.hopsworks.ai and
therefore another level is added to all paths, e.g. `docs.hopsworks.ai/[version-or-alias]`.
Using relative links should not be affected by this, however, building the docs with version
(Option 2) is recommended.
Note: Links and pictures might not resolve properly later on when checking with this build.
The reason for that is that the docs are deployed with versioning on docs.hopsworks.ai and
therefore another level is added to all paths, e.g. `docs.hopsworks.ai/[version-or-alias]`.
Using relative links should not be affected by this, however, building the docs with version
(Option 2) is recommended.

```bash
mkdocs build
# or
mkdocs serve
```
```bash
mkdocs build
# or
mkdocs serve
```

##### Option 2 (Preferred): Build multi-version doc with `mike`

Expand All @@ -118,62 +118,62 @@ On docs.hopsworks.ai we implement the following versioning scheme:

###### Build Instructions

4. For this you can either checkout and make a local copy of the `upstream/gh-pages` branch, where
`mike` maintains the current state of docs.hopsworks.ai, or just build documentation for the branch you are updating:
4. For this you can either checkout and make a local copy of the `upstream/gh-pages` branch, where
`mike` maintains the current state of docs.hopsworks.ai, or just build documentation for the branch you are updating:

Building *one* branch:
Building *one* branch:

Checkout your dev branch with modified docs:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The section below here does not render correctly.

```bash
git checkout [dev-branch]
```
Checkout your dev branch with modified docs:
```bash
git checkout [dev-branch]
```

Generate API docs if necessary:
```bash
python auto_doc.py
```
Generate API docs if necessary:
```bash
python auto_doc.py
```

Build docs with a version and alias
```bash
mike deploy [version] [alias] --update-alias
Build docs with a version and alias
```bash
mike deploy [version] [alias] --update-alias

# for example, if you are updating documentation to be merged to master,
# which will become the new SNAPSHOT version:
mike deploy 3.1.0-SNAPSHOT dev --update-alias
# for example, if you are updating documentation to be merged to master,
# which will become the new SNAPSHOT version:
mike deploy 3.1.0-SNAPSHOT dev --update-alias

# if you are updating docs of the latest stable release branch
mike deploy [version] latest --update-alias
# if you are updating docs of the latest stable release branch
mike deploy [version] latest --update-alias

# if you are updating docs of a previous stable release branch
mike deploy [version]
```
# if you are updating docs of a previous stable release branch
mike deploy [version]
```

If no gh-pages branch existed in your local repository, this will have created it.
If no gh-pages branch existed in your local repository, this will have created it.

**Important**: If no previous docs were built, you will have to choose a version as default to be loaded as index, as follows
**Important**: If no previous docs were built, you will have to choose a version as default to be loaded as index, as follows

```bash
mike set-default [version-or-alias]
```
```bash
mike set-default [version-or-alias]
```

You can now checkout the gh-pages branch and serve:
```bash
git checkout gh-pages
mike serve
```
You can now checkout the gh-pages branch and serve:
```bash
git checkout gh-pages
mike serve
```

You can also list all available versions/aliases:
```bash
mike list
```
You can also list all available versions/aliases:
```bash
mike list
```

Delete and reset your local gh-pages branch:
```bash
mike delete --all
Delete and reset your local gh-pages branch:
```bash
mike delete --all

# or delete single version
mike delete [version-or-alias]
```
# or delete single version
mike delete [version-or-alias]
```

#### Adding new API documentation

Expand All @@ -193,7 +193,7 @@ PAGES = {

Now you can add a template markdown file to the `docs/templates` directory with the name you specified in the auto-doc script. The `new_template.md` file should contain a tag to identify the place at which the API documentation should be inserted:

```
````
## The XYZ package

{{module}}
Expand All @@ -206,7 +206,7 @@ Some extra content here.
```

{{xyz.asd}}
```
````

Finally, run the `auto_doc.py` script, as decribed above, to update the documentation.

Expand Down
Loading
Loading