Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Giovanni Santini

doi:10.1007/978-3-030-21392-3_18

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Giovanni Santini^*

^*Corresponding author for this work

Hong Kong Baptist University

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

7 Citations (Scopus)

Abstract

The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.

Original language	English
Title of host publication	Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings
Editors	Mariana Montiel, Francisco Gomez-Martin, Octavio A. Agustín-Aquino
Publisher	Springer Verlag
Pages	229-235
Number of pages	7
ISBN (Print)	9783030213916
DOIs	https://doi.org/10.1007/978-3-030-21392-3_18
Publication status	Published - 2019
Externally published	Yes
Event	7th International Conference on Mathematics and Computation in Music, MCM 2019 - Madrid, Spain Duration: 18 Jun 2019 → 21 Jun 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11502 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	7th International Conference on Mathematics and Computation in Music, MCM 2019
Country/Territory	Spain
City	Madrid
Period	18/06/19 → 21/06/19

Keywords

Machine Learning
Physical modeling
Synesthesia
Synthesizer
Virtual reality

Access to Document

10.1007/978-3-030-21392-3_18

Cite this

Santini, G. (2019). Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. In M. Montiel, F. Gomez-Martin, & O. A. Agustín-Aquino (Eds.), Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings (pp. 229-235). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11502 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-030-21392-3_18

Santini, Giovanni. / Synesthesizer : Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings. editor / Mariana Montiel ; Francisco Gomez-Martin ; Octavio A. Agustín-Aquino. Springer Verlag, 2019. pp. 229-235 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{b39321e534bb4995b8d810695de52eca,

title = "Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality",

abstract = "The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.",

keywords = "Machine Learning, Physical modeling, Synesthesia, Synthesizer, Virtual reality",

author = "Giovanni Santini",

year = "2019",

doi = "10.1007/978-3-030-21392-3_18",

language = "English",

isbn = "9783030213916",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "229--235",

editor = "Mariana Montiel and Francisco Gomez-Martin and Agust{\'i}n-Aquino, {Octavio A.}",

booktitle = "Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings",

}

Santini, G 2019, Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. in M Montiel, F Gomez-Martin & OA Agustín-Aquino (eds), Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11502 LNAI, Springer Verlag, pp. 229-235, 7th International Conference on Mathematics and Computation in Music, MCM 2019, Madrid, Spain, 18/06/19. https://doi.org/10.1007/978-3-030-21392-3_18

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. / Santini, Giovanni.
Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings. ed. / Mariana Montiel; Francisco Gomez-Martin; Octavio A. Agustín-Aquino. Springer Verlag, 2019. p. 229-235 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11502 LNAI).

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

TY - GEN

T1 - Synesthesizer

T2 - 7th International Conference on Mathematics and Computation in Music, MCM 2019

AU - Santini, Giovanni

PY - 2019

Y1 - 2019

N2 - The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.

AB - The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation;Given its intuitiveness, it might allow even non-experts to explore the possibilities of sound synthesis. The current version has been developed in a Virtual Reality (VR) environment.

KW - Machine Learning

KW - Physical modeling

KW - Synesthesia

KW - Synthesizer

KW - Virtual reality

UR - http://www.scopus.com/inward/record.url?scp=85067865172&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-21392-3_18

DO - 10.1007/978-3-030-21392-3_18

M3 - Conference Proceeding

AN - SCOPUS:85067865172

SN - 9783030213916

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 229

EP - 235

BT - Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings

A2 - Montiel, Mariana

A2 - Gomez-Martin, Francisco

A2 - Agustín-Aquino, Octavio A.

PB - Springer Verlag

Y2 - 18 June 2019 through 21 June 2019

ER -

Santini G. Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality. In Montiel M, Gomez-Martin F, Agustín-Aquino OA, editors, Mathematics and Computation in Music - 7th International Conference, MCM 2019, Proceedings. Springer Verlag. 2019. p. 229-235. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-21392-3_18

Synesthesizer: Physical Modelling and Machine Learning for a Color-Based Synthesizer in Virtual Reality

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this