diff --git a/DESCRIPTION b/DESCRIPTION index 80f8d3b2..61d603b7 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -42,11 +42,8 @@ Suggests: progressr, processx, redux, - rmarkdown, testthat (>= 3.0.0), rush -VignetteBuilder: - knitr Config/testthat/edition: 3 Config/testthat/parallel: false Encoding: UTF-8 diff --git a/R/OptimizerAsyncDesignPoints.R b/R/OptimizerAsyncDesignPoints.R index c8917dad..4a0ca4ff 100644 --- a/R/OptimizerAsyncDesignPoints.R +++ b/R/OptimizerAsyncDesignPoints.R @@ -34,7 +34,7 @@ OptimizerAsyncDesignPoints = R6Class("OptimizerAsyncDesignPoints", param_classes = c("ParamLgl", "ParamInt", "ParamDbl", "ParamFct", "ParamUty"), properties = c("dependencies", "single-crit", "multi-crit"), packages = "rush", - label = "Design Points", + label = "Asynchronous Design Points", man = "bbotk::mlr_optimizers_async_design_points" ) }, diff --git a/README.Rmd b/README.Rmd index 6f4415dd..e2b638c9 100644 --- a/README.Rmd +++ b/README.Rmd @@ -25,39 +25,37 @@ Package website: [release](https://bbotk.mlr-org.com/) | [dev](https://bbotk.mlr *bbotk* is a black-box optimization framework for R. It features highly configurable search spaces via the [paradox](https://github.com/mlr-org/paradox) package and optimizes every user-defined objective function. -The package includes several optimization algorithms e.g. Random Search, Iterated Racing, Bayesian Optimization (in [mlr3mbo](https://github.com/mlr-org/mlr3mbo)) and Hyperband (in [mlr3hyperband](https://github.com/mlr-org/mlr3hyperband)). +The package includes several optimization algorithms e.g. Random Search, Grid Search, Iterated Racing, Bayesian Optimization (in [mlr3mbo](https://github.com/mlr-org/mlr3mbo)) and Hyperband (in [mlr3hyperband](https://github.com/mlr-org/mlr3hyperband)). bbotk is the base package of [mlr3tuning](https://github.com/mlr-org/mlr3tuning), [mlr3fselect](https://github.com/mlr-org/mlr3fselect) and [miesmuschel](https://github.com/mlr-org/miesmuschel). -The package includes the basic building blocks of optimization: - -* `Optimizer`: Objects of this class allow you to optimize an object of the class `OptimInstance`. -* `OptimInstance`: Defines the optimization problem, consisting of an `Objective`, the `search_space`, and a `Terminator`. - All evaluations on the `OptimInstance` will be automatically stored in its own `Archive`. -* `Objective`: Objects of this class contain the objective function. - The class ensures that the objective function is called in the right way and defines, whether the function should be minimized or maximized. -* `Terminator`: Objects of this class control the termination of the optimization independent of the optimizer. - ## Resources -* Package [vignette](https://CRAN.R-project.org/package=bbotk/vignettes/bbotk.html) +There are several sections about black-box optimization in the [mlr3book](https://mlr3book.mlr-org.com). +Often the sections about tuning are also relevant for general black-box optimization. + +* Getting started with [black-box optimization](https://mlr3book.mlr-org.com/chapters/chapter5/advanced_tuning_methods_and_black_box_optimization.html#sec-black-box-optimization). +* An overview of all optimizers and tuners can be found on our [website](https://mlr-org.com/tuners.html). +* Learn about log transformations in the [search space](https://mlr3book.mlr-org.com/chapters/chapter4/hyperparameter_optimization.html#sec-logarithmic-transformations). +* Or more advanced [search space transformations](https://mlr3book.mlr-org.com/chapters/chapter4/hyperparameter_optimization.html#sec-tune-trafo). +* Run [multi-objective optimization](https://mlr3book.mlr-org.com/chapters/chapter5/advanced_tuning_methods_and_black_box_optimization.html#sec-multi-metrics-tuning). +* The [mlr3viz](https://github.com/mlr-org/mlr3viz) package can be used to [visualize](https://mlr-org.com/gallery/technical/2022-12-22-mlr3viz/#tuning-instance) the optimization process. +* Quick optimization with the [`bb_optimize`](https://bbotk.mlr-org.com/reference/bb_optimize.html) function. ## Installation -Install the last release from CRAN: +Install the latest release from CRAN. ```{r eval = FALSE} install.packages("bbotk") ``` -Install the development version from GitHub: +Install the development version from GitHub. ```{r eval = FALSE} -remotes::install_github("mlr-org/bbotk") +pak::pkg_install("mlr-org/bbotk") ``` -## Examples - -### Optimization +## Example ```{r} # define the objective function @@ -76,7 +74,7 @@ codomain = ps( y = p_dbl(tags = "maximize") ) -# create Objective object +# create objective objective = ObjectiveRFun$new( fun = fun, domain = domain, @@ -84,13 +82,10 @@ objective = ObjectiveRFun$new( properties = "deterministic" ) -# Define termination criterion -terminator = trm("evals", n_evals = 10) - -# create optimization instance -instance = OptimInstanceBatchSingleCrit$new( +# initialize instance +instance = oi( objective = objective, - terminator = terminator + terminator = trm("evals", n_evals = 20) ) # load optimizer @@ -106,23 +101,3 @@ instance$result as.data.table(instance$archive) ``` -### Quick optimization with `bb_optimize` - -```{r} -library(bbotk) - -# define the objective function -fun = function(xs) { - c(y1 = - (xs[[1]] - 2)^2 - (xs[[2]] + 3)^2 + 10) -} - -# optimize function with random search -result = bb_optimize(fun, method = "random_search", lower = c(-10, -5), upper = c(10, 5), - max_evals = 100) - -# optimized parameters -result$par - -# optimal outcome -result$value -``` diff --git a/README.md b/README.md index aa6eb963..75550415 100644 --- a/README.md +++ b/README.md @@ -16,51 +16,53 @@ Badge](https://www.r-pkg.org/badges/version-ago/bbotk)](https://cran.r-project.o configurable search spaces via the [paradox](https://github.com/mlr-org/paradox) package and optimizes every user-defined objective function. The package includes several -optimization algorithms e.g. Random Search, Iterated Racing, Bayesian -Optimization (in [mlr3mbo](https://github.com/mlr-org/mlr3mbo)) and -Hyperband (in +optimization algorithms e.g. Random Search, Grid Search, Iterated +Racing, Bayesian Optimization (in +[mlr3mbo](https://github.com/mlr-org/mlr3mbo)) and Hyperband (in [mlr3hyperband](https://github.com/mlr-org/mlr3hyperband)). bbotk is the base package of [mlr3tuning](https://github.com/mlr-org/mlr3tuning), [mlr3fselect](https://github.com/mlr-org/mlr3fselect) and [miesmuschel](https://github.com/mlr-org/miesmuschel). -The package includes the basic building blocks of optimization: - -- `Optimizer`: Objects of this class allow you to optimize an object - of the class `OptimInstance`. -- `OptimInstance`: Defines the optimization problem, consisting of an - `Objective`, the `search_space`, and a `Terminator`. All evaluations - on the `OptimInstance` will be automatically stored in its own - `Archive`. -- `Objective`: Objects of this class contain the objective function. - The class ensures that the objective function is called in the right - way and defines, whether the function should be minimized or - maximized. -- `Terminator`: Objects of this class control the termination of the - optimization independent of the optimizer. - ## Resources -- Package - [vignette](https://CRAN.R-project.org/package=bbotk/vignettes/bbotk.html) +There are several sections about black-box optimization in the +[mlr3book](https://mlr3book.mlr-org.com). Often the sections about +tuning are also relevant for general black-box optimization. + +- Getting started with [black-box + optimization](https://mlr3book.mlr-org.com/chapters/chapter5/advanced_tuning_methods_and_black_box_optimization.html#sec-black-box-optimization). +- An overview of all optimizers and tuners can be found on our + [website](https://mlr-org.com/tuners.html). +- Learn about log transformations in the [search + space](https://mlr3book.mlr-org.com/chapters/chapter4/hyperparameter_optimization.html#sec-logarithmic-transformations). +- Or more advanced [search space + transformations](https://mlr3book.mlr-org.com/chapters/chapter4/hyperparameter_optimization.html#sec-tune-trafo). +- Run [multi-objective + optimization](https://mlr3book.mlr-org.com/chapters/chapter5/advanced_tuning_methods_and_black_box_optimization.html#sec-multi-metrics-tuning). +- The [mlr3viz](https://github.com/mlr-org/mlr3viz) package can be + used to + [visualize](https://mlr-org.com/gallery/technical/2022-12-22-mlr3viz/#tuning-instance) + the optimization process. +- Quick optimization with the + [`bb_optimize`](https://bbotk.mlr-org.com/reference/bb_optimize.html) + function. ## Installation -Install the last release from CRAN: +Install the latest release from CRAN. ``` r install.packages("bbotk") ``` -Install the development version from GitHub: +Install the development version from GitHub. ``` r -remotes::install_github("mlr-org/bbotk") +pak::pkg_install("mlr-org/bbotk") ``` -## Examples - -### Optimization +## Example ``` r # define the objective function @@ -79,7 +81,7 @@ codomain = ps( y = p_dbl(tags = "maximize") ) -# create Objective object +# create objective objective = ObjectiveRFun$new( fun = fun, domain = domain, @@ -87,13 +89,10 @@ objective = ObjectiveRFun$new( properties = "deterministic" ) -# Define termination criterion -terminator = trm("evals", n_evals = 10) - -# create optimization instance -instance = OptimInstanceBatchSingleCrit$new( +# initialize instance +instance = oi( objective = objective, - terminator = terminator + terminator = trm("evals", n_evals = 20) ) # load optimizer @@ -103,16 +102,16 @@ optimizer = opt("gensa") optimizer$optimize(instance) ``` - ## x1 x2 x_domain y - ## 1: 2.0452 -2.064743 9.123252 + ## x1 x2 x_domain y + ## 1: 2 -3 10 ``` r # best performing configuration instance$result ``` - ## x1 x2 x_domain y - ## 1: 2.0452 -2.064743 9.123252 + ## x1 x2 x_domain y + ## 1: 2 -3 10 ``` r # all evaluated configuration @@ -120,42 +119,14 @@ as.data.table(instance$archive) ``` ## x1 x2 y timestamp batch_nr x_domain_x1 x_domain_x2 - ## 1: -4.689827 -1.278761 -37.716445 2024-06-21 09:34:39 1 -4.689827 -1.278761 - ## 2: -5.930364 -4.400474 -54.851999 2024-06-21 09:34:39 2 -5.930364 -4.400474 - ## 3: 7.170817 -1.519948 -18.927907 2024-06-21 09:34:39 3 7.170817 -1.519948 - ## 4: 2.045200 -1.519948 7.807403 2024-06-21 09:34:39 4 2.045200 -1.519948 - ## 5: 2.045200 -2.064742 9.123250 2024-06-21 09:34:39 5 2.045200 -2.064742 - ## 6: 2.045200 -2.064742 9.123250 2024-06-21 09:34:39 6 2.045200 -2.064742 - ## 7: 2.045201 -2.064742 9.123250 2024-06-21 09:34:39 7 2.045201 -2.064742 - ## 8: 2.045199 -2.064742 9.123250 2024-06-21 09:34:39 8 2.045199 -2.064742 - ## 9: 2.045200 -2.064741 9.123248 2024-06-21 09:34:39 9 2.045200 -2.064741 - ## 10: 2.045200 -2.064743 9.123252 2024-06-21 09:34:39 10 2.045200 -2.064743 - -### Quick optimization with `bb_optimize` - -``` r -library(bbotk) - -# define the objective function -fun = function(xs) { - c(y1 = - (xs[[1]] - 2)^2 - (xs[[2]] + 3)^2 + 10) -} - -# optimize function with random search -result = bb_optimize(fun, method = "random_search", lower = c(-10, -5), upper = c(10, 5), - max_evals = 100) - -# optimized parameters -result$par -``` - - ## x1 x2 - ## 1: -7.982537 4.273021 - -``` r -# optimal outcome -result$value -``` - - ## y1 - ## -142.5479 + ## 1: -4.689827 -1.278761 -37.716445 2024-08-13 17:52:54 1 -4.689827 -1.278761 + ## 2: -5.930364 -4.400474 -54.851999 2024-08-13 17:52:54 2 -5.930364 -4.400474 + ## 3: 7.170817 -1.519948 -18.927907 2024-08-13 17:52:54 3 7.170817 -1.519948 + ## 4: 2.045200 -1.519948 7.807403 2024-08-13 17:52:54 4 2.045200 -1.519948 + ## 5: 2.045200 -2.064742 9.123250 2024-08-13 17:52:54 5 2.045200 -2.064742 + ## --- + ## 16: 2.000000 -3.000000 10.000000 2024-08-13 17:52:54 16 2.000000 -3.000000 + ## 17: 2.000001 -3.000000 10.000000 2024-08-13 17:52:54 17 2.000001 -3.000000 + ## 18: 1.999999 -3.000000 10.000000 2024-08-13 17:52:54 18 1.999999 -3.000000 + ## 19: 2.000000 -2.999999 10.000000 2024-08-13 17:52:54 19 2.000000 -2.999999 + ## 20: 2.000000 -3.000001 10.000000 2024-08-13 17:52:54 20 2.000000 -3.000001 diff --git a/vignettes/bbotk.Rmd b/vignettes/bbotk.Rmd deleted file mode 100644 index 966f8e86..00000000 --- a/vignettes/bbotk.Rmd +++ /dev/null @@ -1,271 +0,0 @@ ---- -title: "bbotk: A brief introduction" -author: "Jakob Richter" -date: "`r Sys.Date()`" -output: rmarkdown::html_vignette -vignette: > - %\VignetteIndexEntry{bbotk: A brief introduction} - %\VignetteEngine{knitr::rmarkdown} - %\VignetteEncoding{UTF-8} ---- - -```{r setup, include = FALSE} -set.seed(1) -knitr::opts_chunk$set( - collapse = TRUE, - comment = "#>" -) -``` - -# Overview - -The main goal of the *black box optimization toolkit* (`bbotk`) is to provide a common framework for optimization for other packages. -Therefore `bbotk` includes the following *R6* classes that can be used in a variety of optimization scenarios. - -- `Optimizer`: Objects of this class allow you to optimize an object of the class `OptimInstance`. -- `OptimInstance`: Defines the optimization problem, consisting of an `Objective`, the `search_space` and a `Terminator`. - All evaluations on the `OptimInstance` will be automatically stored in its own `Archive`. -- `Objective`: Objects of this class contain the objective function. - The class ensures that the objective function is called in the right way and defines, whether the function should be minimized or maximized. -- `Terminator`: Objects of this class control the termination of the optimization independent of the optimizer. - -As `bbotk` also includes some basic optimizers and can be used on its own. -The registered optimizers can be queried as follows: - -```{r} -library(bbotk) -opts() -``` - -This Vignette will show you how to use the `bbotk`-classes to solve a simple optimization problem. -Furthermore, you will learn how to - -- construct your `Objective`. -- define your optimization problem in an `OptimInstance` -- define a restricted `search_space`. -- define the logging threshold. -- access the `Archive` of evaluated function calls. - -## Use `bbotk` to optimize a function - -In the following we will use `bbotk` to minimize this function: - -```{r} -fun = function(xs) { - c(y = - (xs[[1]] - 2)^2 - (xs[[2]] + 3)^2 + 10) -} -``` - -First we need to wrap `fun` inside an `Objective` object. -For functions that expect a list as input we can use the `ObjectiveRFun` class. -Additionally, we need to specify the domain, i.e. the space of x-values that the function accepts as an input. -Optionally, we can define the co-domain, i.e. the output space of our objective function. -This is only necessary if we want to deviate from the default which would define the output to be named *y* and be minimized. -Such spaces are defined using the package [`paradox`](https://cran.r-project.org/package=paradox). - -```{r} -library(paradox) - -domain = ps( - x1 = p_dbl(-10, 10), - x2 = p_dbl(-5, 5) -) -codomain = ps( - y = p_dbl(tags = "maximize") -) -obfun = ObjectiveRFun$new( - fun = fun, - domain = domain, - codomain = codomain, - properties = "deterministic" # i.e. the result always returns the same result for the same input. -) -``` - -In the next step we decide when the optimization should stop. -We can list all available terminators as follows: - -```{r} -trms() -``` - -The termination should stop, when it takes longer then 10 seconds or when 20 evaluations are reached. - -```{r} -terminators = list( - evals = trm("evals", n_evals = 20), - run_time = trm("run_time") -) -terminators -``` - -We have to correct the default of `secs=30` by setting the `values` in the `param_set` of the terminator. - -```{r} -terminators$run_time$param_set$values$secs = 10 -``` - -We have created `Terminator` objects for both of our criteria. -To combine them we use the *combo* `Terminator`. - -```{r} -term_combo = TerminatorCombo$new(terminators = terminators) -``` - -Before we finally start the optimization, we have to create an `OptimInstance` that contains also the `Objective` and the `Terminator`. - -```{r} -instance = OptimInstanceBatchSingleCrit$new(objective = obfun, terminator = term_combo) -instance -``` - -Note, that `OptimInstance(SingleCrit/MultiCrit)$new()` also has an optional `search_space` argument. -It can be used if the `search_space` is only a subset of `obfun$domain` or if you want to apply transformations. -More on that later. - -Finally, we have to define an `Optimizer`. -As we have seen above, that we can call `opts()` to list all available optimizers. -We opt for evolutionary optimizer, from the `GenSA` package. - -```{r} -optimizer = opt("gensa") -optimizer -``` - -To start the optimization we have to call the `Optimizer` on the `OptimInstance`. - -```{r} -optimizer$optimize(instance) -``` - -Note, that we did not specify the termination inside the optimizer. -`bbotk` generally sets the termination of the optimizers to never terminate and instead breaks the code internally as soon as a termination criterion is fulfilled. - -The results can be queried from the `OptimInstance`. - -```{r} -# result as a data.table -instance$result -# result as a list that can be passed to the Objective -instance$result_x_domain -# result outcome -instance$result_y -``` - -You can also access the whole history of evaluated points. - -```{r} -as.data.table(instance$archive) -``` - -### Search Space Transformations - -If the domain of the `Objective` is complex, it is often necessary to define a simpler *search space* that can be handled by the `Optimizer` and to define a transformation that transforms the value suggested by the optimizer to a value of the *domain* of the `Objective`. - -Reasons for transformations can be: - -1. The objective is more sensitive to changes of small values than to changes of bigger values of a certain parameter. - E.g. we could suspect that for a parameter `x3` the change from `0.1` to `0.2` has a similar effect as the change of `100` to `200`. -2. Certain restrictions are known, i.e. the sum of three parameters is supposed to be 1. -3. many more... - -In the following we will look at an example for 2.) - -We want to construct a box with the maximal volume, with the restriction that `h+w+d = 1`. -For simplicity we define our problem as a minimization problem. - -```{r} -fun_volume = function(xs) { - c(y = - (xs$h * xs$w * xs$d)) -} -domain = ps( - h = p_dbl(lower = 0), - w = p_dbl(lower = 0), - d = p_dbl(lower = 0) -) -obj = ObjectiveRFun$new( - fun = fun_volume, - domain = domain -) -``` - -We notice, that our optimization problem has three parameters but due to the restriction it only the degree of freedom 2. -Therefore we only need to optimize two parameters and calculate `h`, `w`, `d` accordingly. - -```{r} -search_space = ps( - h = p_dbl(lower = 0, upper = 1), - w = p_dbl(lower = 0, upper = 1), - .extra_trafo = function(x, param_set){ - x = unlist(x) - x["d"] = 2 - sum(x) # model d in dependence of h, w - x = x/sum(x) # ensure that h+w+d = 1 - as.list(x) - } -) -``` - -Instead of the domain of the `Objective` we now use our constructed `search_space` that includes the `trafo` for the `OptimInstance`. - -```{r} -inst = OptimInstanceBatchSingleCrit$new( - objective = obj, - search_space = search_space, - terminator = trm("evals", n_evals = 30) -) -optimizer = opt("gensa") -lg = lgr::get_logger("bbotk")$set_threshold("warn") # turn off console output -optimizer$optimize(inst) -lg = lgr::get_logger("bbotk")$set_threshold("info") # turn on console output -``` -The optimizer only saw the search space during optimization and returns the following result: - -```{r} -inst$result_x_search_space -``` -Internally, the `OptimInstance` transformed these values to the *domain* so that the result for the `Objective` looks as follows: - -```{r} -inst$result_x_domain -obj$eval(inst$result_x_domain) -``` - -### Notes on the optimization archive - -The following is just meant for *advanced* readers. -If you want to evaluate the function outside of the optimization but have the result stored in the `Archive` you can do so by resetting the termination and call `$eval_batch()`. - -```{r} -library(data.table) - -inst$terminator = trm("none") -xvals = data.table(h = c(0.6666, 0.6667), w = c(0.6666, 0.6667)) -inst$eval_batch(xdt = xvals) -tail(as.data.table(instance$archive)) -``` -However, this does not update the result. -You could set the result by calling `inst$assign_result()` but this should be handled by the optimizer. -Another way to get the best point is the following: - -```{r} -inst$archive$best() -``` -Note, that this method just looks for the best outcome and returns the according row from the archive. -For stochastic optimization problems this is overly optimistic and leads to biased results. -For this reason the optimizer can use advanced methods to determine the result and set it itself. - - -```{r, eval = FALSE, include = FALSE} -# TODO: Write the following sections: - -## Implementing your own Objective - -### Storing extra output in the archive - -## Implementing your own Optimizer - -### Storing extra tuner information in the archive - -## Implement your own Terminator - -# TODO: Fix intro after more sections are added -```