Computational Vegetation Group

Computational Vegetation Group focuses on digital twins of plants, particularly on digital tree models. Building on Dr. Benes’s 25 years of experience, we research the shape, structure, and function of vegetation. We study the plant’s shape, development over time, and interaction with its environment (light, water, wind, temperature). Most of the work has been done in collaboration with our colleagues and past members, listed below.

3D Vegetation Reconstruction - Digital Twins

Together with our collaborators, we have developed algorithms for AI-based 3D tree reconstruction from photographs, point clouds, inverse procedural models of trees as L-systems, and developmental models programmed as simulation systems. The input to this problem is an existing tree represented as an image, point cloud, or geometry. The goal is to find a functional-structural plant model (digital twin) that can be used in simulating “what-if” scenarios. The figure below from inverse procedural modeling of trees shows such a scenario. The input tree on the left is analyzed, and the simulation parameters that can reproduce it are found. Having the digital twin, the tree can be regrown in a different environment (right).

See the video of the reconstruction from images:

Crops in Silico and in Silico Plants

Our paper from 2020 describes the importance of multiscale computational models for crops, an initiative called Crops in Silico.

Together with our collaborators, we have developed algorithms that allow for finding genetic traits and 3D reconstruction for phenotyping facilities. In particular, our voxel carving algorithm for sorghum has been used to identify the gene that controls its branching angle.

The figure below shows the 3D reconstruction of maize, where (a) is the side and top view of a procedurally generated 3D model of a maize plant, (b) shows a voxelized version of the original 3D model, and (c) a reconstructed 3D model generated using the voxel carving.

A recent paper discusses current challenges and future of agricultural genomes to phenomes in the USA. It is a result of a three day workshop at Agricultural Genome to Phenome Initiative.

Deep Neural Models of Vegetation

We have developed algorithms that encode tree structure and behavior as a deep neural model. In particular, it encodes tree branchlets as a latent space that can be then used to create novel trees or reconstruct existing from point clouds.

Vegetation Simulation and Modeling

The video below shows the tree’s developmental and immediate response to wind. This model captures the physics response of the tree shape to the long and short-term influence of wind.

The developmental model can show climbing plants (Honorable mention from the Best Papers Committee Eurographics 2017), and there was also a paper in 2002 on the same topic.

The tree environmental response of the tree can be estimated from its geometry. The response can then be used to simulate the tree response to a different environment interactively, an approach we called plastic trees. The reconstructed model is environmentally sensitive and responds to light or obstacles, as shown in this video.

Ecosystems and Urban Forests

Entire landscapes can be authored by combining ecosystems and erosion, as shown in this paper. An urban forest can also be considered a special case of an ecosystem, as shown in this, where we find tree location, and we generate an inverse procedural model showing where and how to plant the trees. An early paper considered the urban forest as an unbalanced ecosystem, where the disturbance is caused by pruning, removing unwanted plants, planting in order, etc. Several earlier works studied the effect of virtual agents on ecosystems. An interesting approach is to simulate ecosystems interactively and even while considering animals

We have also developed the first algorithm that finds the location of trees in an urban environment (urban forest) that has also been used and extended for tree counting. The figure below shows the result of the procedural tree positioning in a hypothetical scenario of Central Park in New York. An older paper also shows urban forests as affected by how people care about the trees.

One of the critical aspects is the plant development. Our developmental model allows for creating windy trees that show the urban forest under the influence of wind.

Orchards and IMApple

Together with our collaborators, we have developed a source-sink developmental model for Golden Delicious apple trees. The trees consume water and use light for photosynthesis. The result is growth and apples on the trees, as shown in this image:

We also studied the effect of pruning on orchards and co-developed an automatic pruning optimization algorithm that finds what branch should be removed to maximize light intake of a tree.

Validation

One of the open problems is the validation of the computer-generated models. Together with our collaborators, we have developed the first perceptual model called ICTree for assessing the perceived realism of tree models. An AI-based model was trained on one million user evaluations and can predict the perceived quality of the tree model either from its geometry or an image. An example below shows an editing session and the feedback on the metric.

Current Members

Jae Joong Lee

Bosheng Li

Ian Andrew Ostermann

Zhiquan Wang

Xiaochen Zhou

Past Members

Javier Abdul Cordoba (2003)


Institution:

Michel Abdul-Massih, Ph.D. (2014)


Institution: Electronic Arts

Marek Fiser, M.S. (2015)


Institution: Google Inc

Alejandro Guayaquil, Ph.D. (2014)


Institution: Siemens Inc

Mathieu Gaillard, Ph.D. (2022)


Institution: Adobe Research

David Hrusa, M.S. (2021)


Institution:

Hao Kang, Ph.D. (2019)


Institution: Wormplex AI Research

Vojtech Krs, Ph.D. (2019)


Institution: Adobe Research

Juan Miguel Soto, M.S.


Institution:

Eric Millan, M.S.


Institution:

Ondrej Stava, Ph.D. (2012)


Institution: Google Inc

Juraj Vanek, Ph.D. (2014)


Institution: Relativity Space

Collaborators

Martin Čadík

Marie-Paule Cani

Guillaume Cordonnier

Oliver Deussen

Pierre Ecormier-Nocca

Arnaud Emilien

Jianwei Guo

Rado Gazo

Torsten Hädrich

John Hart

Eva Haviarova

Peter Hirst

Hui Huang

Jacek Kaluzny

Jonathan Klein

Stefan Kohek

Simon Kolmanič

Julian Kratt

Yanchao Liu

Amy Marshall-Collon

Radomir Mech

Chenyong Miao

Dominik Michels

Till Niese

Wojteck Palubicki

Soeren Pirk

Tomáš Polášek

James Schnable

Damjan Strnad

Michael Tross

Logan Wells

Fanyou Wu

Borut Zalik

Xiaopen Zhang

Computational Vegetation Publications

2025

    2024

    1. Lee, J. J., Li, B., Beery, S., Huang, J., Fei, S., Yeh, R. A., & Benes, B. (2024). Tree-D Fusion: Simulation-Ready Tree Dataset from Single Images with Diffusion Priors. Computer Vision – ECCV 2024, 439–460. https://doi.org/https://doi.org/10.1007/978-3-031-72940-9_25
    2. Li, B., Schwarz, N. A., Pałubicki, W., Pirk, S., & Benes, B. (2024). Interactive Invigoration: Volumetric Modeling of Trees with Strands. ACM Trans. Graph., 43(4). https://doi.org/10.1145/3658206
    3. Lee, J. J., Li, B., & Benes, B. (2024). Latent L-Systems: Transformer-Based Tree Generator. ACM Trans. Graph., 43(1). https://doi.org/10.1145/3627101
    4. Li, Y., Liu, Z., Benes, B., Zhang, X., & Guo, J. (2024). SVDTree: Semantic Voxel Diffusion for Single Image Tree Reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4692–4702.
    5. Tuggle, C. K., Clarke, J. L., Murdoch, B. M., Lyons, E., Scott, N. M., Benes, B., Campbell, J. D., Chung, H., Daigle, C. L., Choudhury, S. D., Dekkers, J. C. M., Dorea, J. R. R., Ertl, D. S., Feldman, M., Fragomeni, B. O., Fulton, J. E., Guadagno, C. R., Hagen, D. E., Hess, A. S., … Schnable, P. S. (2024). Current challenges and future of agricultural genomes to phenomes in the USA. Genome Biology, 25(8). https://doi.org/10.1186/s13059-023-03155-w
    6. Zarei, A., Li, B., Schnable, J. C., Lyons, E., Pauli, D., Barnard, K., & Benes, B. (2024). PlantSegNet: 3D point cloud instance segmentation of nearby plant organs with identical semantics. Computers and Electronics in Agriculture, 221, 108922. https://doi.org/https://doi.org/10.1016/j.compag.2024.108922
    7. Ostermann, I., Benes, B., Gaillard, M., Li, B., Davis, J., Grove, R., Shrestha, N., Tross, M. C., & Schnable, J. C. (2024). Sorghum segmentation and leaf counting using in silico trained deep neural model. The Plant Phenome Journal, 7(1), e70002. https://doi.org/https://doi.org/10.1002/ppj2.70002
    8. Warner, C., Wu, F., Gazo, R., Benes, B., Kong, N., & Fei, S. (2024). CentralBark Image Dataset and Tree Species Classification Using Deep Learning. Algorithms, 17(5). https://doi.org/10.3390/a17050179

    2023

    1. Rhizomorph: The Coordinated Function of Shoots and Roots. (2023). ACM Transaction on Graphics, 42(4). https://doi.org/https://doi.org/10.1145/3592145
      Replicability Stamp https://www.replicabilitystamp.org/#https-github-com-edisonlee0212-rhizomorph
    2. Zhou, X., Li, B., Benes, B., Fei, S., & Pirk, S. (2023). DeepTree: Modeling Trees with Situated Latents. IEEE Transactions on Visualization & Computer Graphics, 01, 1–14. https://doi.org/10.1109/TVCG.2023.3307887
    3. Firoze, A., Wingren, C., Yeh, R. A., Benes, B., & Aliaga, D. (2023). Tree Instance Segmentation With Temporal Contour Graph. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2193–2202.

    2022

    1. Niese, T., Pirk, S., Albrecht, M., Benes, B., & Deussen, O. (2022). Procedural Urban Forestry. ACM Trans. Graph., 41(2). https://doi.org/10.1145/3502220
    2. Firoze, A., Benes, B., & Aliaga, D. (2022). Urban tree generator: spatio-temporal and generative deep learning for urban tree localization and modeling. The Visual Computer. https://doi.org/10.1007/s00371-022-02526-x

    2021

    1. Polasek, T., Hrusa, D., Benes, B., & Cadik, M. (2021). ICTree: Automatic Perceptual Metrics for Tree Models. ACM Transaction on Graphics, 40(6), 1–15. https://doi.org/10.1145/3478513.3480519
    2. Li, B., Kałużny, J., Klein, J., Michels, D. L., Pałubicki, W., Benes, B., & Pirk, S. (2021). Learning to Reconstruct Botanical Trees from Single Images. ACM Transaction on Graphics, 40(6), 1–15. https://doi.org/10.1145/3478513.3480525
    3. Liu, Y., Guo, J., Benes, B., Deussen, O., Zhang, X., & Huang, H. (2021). TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction. ACM Transaction on Graphics, 40(6), 1–16. https://doi.org/10.1145/3478513.3480486
    4. Ecormier-Nocca, P., Cordonnier, G., Carrez, P., Moigne, A.-marie, Memari, P., Benes, B., & Cani, M.-P. (2021). Authoring Consistent Landscapes with Flora and Fauna. ACM Transactions on Graphics, 40(4). https://doi.org/10.1145/3450626.3459952
    5. Tross, M. C., Gaillard, M., Zweiner, M., Miao, C., Grove, R. J., Li, B., Benes, B., & Schnable, J. C. (2021). 3D reconstruction identifies loci linked to variation in angle of individual sorghum leaves. PeerJ, 9:e12628(1). https://doi.org/10.7717/peerj.12628
    6. Wu, F., Gazo, R., Benes, B., & Haviarova, E. (2021). Deep BarkID: a portable tree bark identification system by knowledge distillation. European Journal of Forest Research. https://doi.org/10.1007/s10342-021-01407-7
    7. Wu, F., Gazo, R., Haviarova, E., & Benes, B. (2021). Wood identification based on longitudinal section images by using deep learning. Wood Science and Technology, 55, 553–563. https://doi.org/10.1007/s00226-021-01261-1
    8. Kolmanič, S., Strnad, D., Kohek, Š., Benes, B., Hirst, P., & Žalik, B. (2021). An algorithm for automatic dormant tree pruning. Applied Soft Computing, 99, 106931. https://doi.org/10.1016/j.asoc.2020.106931

    2020

    1. Guo, J., Jiang, H., Benes, B., Deussen, O., Zhang, X., Lischinski, D., & Huang, H. (2020). Inverse Procedural Modeling of Branching Structures by Inferring L-Systems. ACM Trans. Graph., 39(5). https://doi.org/10.1145/3394105
    2. Benes, B., Guan, K., Lang, M., Long, S. P., Lynch, J. P., Marshall-Colón, A., Peng, B., Schnable, J., Sweetlove, L. J., & Turk, M. J. (2020). Multiscale computational models can guide experimentation and targeted measurements for crop improvement. The Plant Journal, 103(1), 21–31. https://doi.org/10.1111/tpj.14722
    3. Strnad, D., Štefan Kohek, Benes, B., Kolmanič, S., & Žalik, B. (2020). A framework for multi-objective optimization of virtual tree pruning based on growth simulation. Expert Systems with Applications, 162, 113792. https://doi.org/https://doi.org/10.1016/j.eswa.2020.113792
    4. Gaillard, M., Miao, C., Schnable, J. C., & Benes, B. (2020). Voxel carving-based 3D reconstruction of sorghum identifies genetic determinants of light interception efficiency. Plant Direct, 4(10), e00255. https://doi.org/10.1002/pld3.255
    5. Gaillard, M., Miao, C., Schnable, J., & Benes, B. (2020). Sorghum Segmentation by Skeleton Extraction. In A. Bartoli & A. Fusiello (Eds.), Computer Vision – ECCV 2020 Workshops (pp. 296–311). Springer International Publishing. https://doi.org/ 10.1007/978-3-030-65414-6_21
    6. Gazo, R., Vanek, J., Abdul-Massih, M., & Benes, B. (2020). A fast pith detection for computed tomography scanned hardwood logs. Computers and Electronics in Agriculture, 170, 105107. https://doi.org/https://doi.org/10.1016/j.compag.2019.105107

    2019

    1. Gaillard, M., Benes, B., Guérin, E., Galin, E., Rohmer, D., & Cani, M.-P. (2019). Dendry: A Procedural Model for Dendritic Patterns. Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 16:1–16:9. https://doi.org/10.1145/3306131.3317020

    2018

    1. Gazo, R., Wells, L., Krs, V., & Benes, B. (2018). Validation of automated hardwood lumber grading system. Computers and Electronics in Agriculture, 155, 496–500. https://doi.org/https://doi.org/10.1016/j.compag.2018.06.041
    2. Wells, L., Gazo, R., Re, R. D., Krs, V., & Benes, B. (2018). Defect detection performance of automated hardwood lumber grading system. Computers and Electronics in Agriculture, 155, 487–495. https://doi.org/https://doi.org/10.1016/j.compag.2018.09.025

    2017

    1. Cordonnier, G., Galin, E., Gain, J., Benes, B., Guérin, E., Peytavie, A., & Cani, M.-P. (2017). Authoring Landscapes by Combining Ecosystem and Terrain Erosion Simulation. ACM Trans. Graph., 36(4), 134:1–134:12. https://doi.org/10.1145/3072959.3073667
    2. Hädrich, T., Benes, B., Deussen, O., & Pirk, S. (2017). Interactive modeling and authoring of climbing plants. Computer Graphics Forum, 36(2), 49–61. https://doi.org/10.1111/cgf.13106
      Honorable mention from the Best Papers Committee
    3. Marshall-Colon, A., Long, S. P., Allen, D. K., Allen, G., Beard, D. A., Benes, B., von Caemmerer, S., Christensen, A. J., Cox, D. J., Hart, J. C., Hirst, P. M., Kannan, K., Katz, D. S., Lynch, J. P., Millar, A. J., Panneerselvam, B., Price, N. D., Prusinkiewicz, P., Raila, D., … Zhu, X.-G. (2017). Crops In Silico: Generating Virtual Crops Using an Integrative and Multi-scale Modeling Platform. Frontiers in Plant Science, 8, 786. https://doi.org/10.3389/fpls.2017.00786
    4. Fišer, M., Ravi, J., Benes, B., Shi, B., & Hirst, P. (2017). IMapple: a source-sink developmental model for’Golden Delicious’ apple trees. Acta Hortic, 51–60. https://doi.org/doi.org/10.17660/ActaHortic.2017.1160.8

    2016

    1. Pirk, S., Benes, B., Ijiri, T., Li, Y., Deussen, O., Chen, B., & Měch, R. (2016). Modeling Plant Life in Computer Graphics. ACM SIGGRAPH 2016 Courses, 18:1–18:180. https://doi.org/10.1145/2897826.2927332
    2. Kang, H., Fiser, M., Shi, B., Sheibani, F., Hirst, P., & Benes, B. (2016). IMapple - Functional structural model of apple trees. 2016 IEEE International Conference on Functional-Structural Plant Growth Modeling, Simulation, Visualization and Applications (FSPMA), 90–97. https://doi.org/10.1109/FSPMA.2016.7818293

    2015

    1. Emilien, A., Vimont, U., Cani, M.-P., Poulin, P., & Benes, B. (2015). WorldBrush: Interactive Example-Based Synthesis of Procedural Virtual Worlds. ACM Trans. Graph., 34(4). https://doi.org/10.1145/2766975
    2. Kratt, J., Spicker, M., Guayaquil, A., Fiser, M., Pirk, S., Deussen, O., Hart, J. C., & Benes, B. (2015). Woodification: User-Controlled Cambial Growth Modeling. Computer Graphics Forum, 34(2), 361–372. https://doi.org/10.1111/cgf.12566

    2014

    1. Pirk, S., Niese, T., Hädrich, T., Benes, B., & Deussen, O. (2014). Windy Trees: Computing Stress Response for Developmental Tree Models. ACM Trans. Graph., 33(6). https://doi.org/10.1145/2661229.2661252
    2. Šťava, O., Pirk, S., Kratt, J., Chen, B., Měch, R., Deussen, O., & Benes, B. (2014). Inverse Procedural Modelling of Trees. Computer Graphics Forum, 33(6), 118–131. https://doi.org/10.1111/cgf.12282

    2012

    1. Pirk, S., Stava, O., Kratt, J., Said, M. A. M., Neubert, B., Měch Radomı́r, Benes, B., & Deussen, O. (2012). Plastic Trees: Interactive Self-Adapting Botanical Tree Models. ACM Trans. Graph., 31(4). https://doi.org/10.1145/2185520.2185546

    2011

    1. Benes, B., Massih, M. A., Jarvis, P., Aliaga, D. G., & Vanegas, C. A. (2011). Urban Ecosystem Design. Symposium on Interactive 3D Graphics and Games, 167–174. https://doi.org/10.1145/1944745.1944773

    2010

    1. Št’ava, O., Benes, B., Měch, R., Aliaga, D. G., & Krištof, P. (2010). Inverse Procedural Modeling by Automatic Generation of L-systems. Computer Graphics Forum, 29(2), 665–674. https://doi.org/10.1111/j.1467-8659.2009.01636.x

    2009

    1. Benes, B., Andrysco, N., & Št’ava, O. (2009). Interactive Modeling of Virtual Ecosystems. Proceedings of the Fifth Eurographics Conference on Natural Phenomena, 9–16. https://doi.org/10.5555/2381692.2381694

    2004

    1. Benes, B., & Guerrero, J. M. S. (2004). Clustering in virtual plant ecosystems. WSCG SHORT Communication Papers Proceedings.

    2003

    1. Benes, B., & Cordóba, J. A. (2003). Modeling virtual gardens by autonomous procedural agents. Proceedings of Theory and Practice of Computer Graphics, 2003., 58–65. https://doi.org/10.1109/TPCG.2003.1206931
    2. Benes, B., & Espinosa, E. D. (2003). Modeling virtual ecosystems with the proactive guidance of agents. Computer Animation and Social Agents, 2003. 16th International Conference On, 126–131. https://doi.org/10.1109/CASA.2003.1199313
    3. Benes, B., Cordoba, J. A., & Soto, J. M. (2003). Interacting agents with memory in virtual ecosystems. Journal of WSCG.

    2002

    1. Beneš, B. (2002). A stable modeling of large plant ecosystems. Proceedings of the International Conference on Computer Vision and Graphics, 94–101.
    2. Benes, B., & Millán, E. U. (2002). Virtual Climbing Plants Competing for Space. Computer Animation, 33. https://doi.org/10.1109/CA.2002.1017504

    Older

    1. Benes, B. (1997). Visual Model of Plant Development with Respect to Influence of Light. In D. Thalmann & M. van de Panne (Eds.), Computer Animation and Simulation ’97 (pp. 125–136). Springer Vienna. https://doi.org/10.1007/978-3-7091-6874-5_9
    2. Benes, B. (1996). An efficient estimation of light in simulation of plant development. In R. Boulic & G. Hégron (Eds.), Computer Animation and Simulation ’96 (pp. 153–165). Springer Vienna. https://doi.org/10.1007/978-3-7091-7486-9_11
    3. Benes, B. (1998). Skylight approximation for simulation of plant development. Proceedings. 1998 IEEE Conference on Information Visualization. An International Conference on Computer Visualization and Graphics (Cat. No. 98TB100246), 146–150. https://doi.org/10.1109/IV.1998.694212

    Sponsors

    2024-2027

    National Science Foundation, (PI) 2417510

    Collaborative Research IIBRP VR-Bio-Talk VR Voice-Controlled Visual Analytics Platform for Plant Digital Twins


    2024-2027

    National Science Foundation, (PI) 2412928

    Collaborative Research TRTech-PGR Digital Ideotype for Optimal Canopy Architecture


    2024-2027

    National Institute of Food and Agriculture, (co-PI) 1032382

    Optimizing Soybean Canopy Architecture for Efficient Light and Water Use Under Climate Change


    2024-2027

    National Institute of Food and Agriculture, (co-PI) 1032672

    Improving Forest Management Through Automated Measurement of Tree Geometry


    2021-2021

    Agricultural Genome to Phenome Initiative, (PI)

    Optimizing 3D Canopy Architecture for Better Crops,


    2019-2022

    FFAR

    Crops in Silico (see the news here)


    2019-2022

    National Science Foundation, (co-PI) 10001387

    CHS Small Functional Proceduralization of 3D Geometric Models


    2017-2019

    National Science Foundation, (PI) 1608762

    Collaborative Research-Inverse Procedural Material Modeling for Battery Design,


    2015-2016

    Intel, (PI)

    Enhancing Computer Graphics Education with Many Integrated Core Computing


    2014-2016

    Ford Inc, (co-PI)

    Tomographic data reconstruction,


    2014-2015

    Siemens Inc, (PI)

    Software analysis for 3-D printing


    2013-2015

    United States Department of Agriculture(co-PI)

    Integrating Spatial Educational Experiences (Isee) – Mapping a New Approach to Teaching and Learning Soil Science


    2011-2014

    NASA (CO-PI)

    A Global High-Resolution Fossil Fuel CO2 Inventory Built From Assimilation of in Situ and Remotely-Sensed Datasets to Advance Satellite Greenhouse Gas Detection Support Systems


    2010-2013

    National Science Foundation (CO-PI)

    Integrating Behavioral, Geometrical and Graphical Modeling to Simulate and Visualize Urban Areas


    2010-present

    Adobe Inc (PI)

    Procedural Modeling


    2008-2010

    United States Department of Agriculture(co-PI)

    Integrating Spatial Education Experience (ISEE) into Crop, Soil, and Environmental Science Curricula